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Abstract —The Internet is inherently a multipath network—for 
an underlying network with only a single path connecting various 
nodes would have been debilitatingly fragile. Unfortunately, 
traditional Internet technologies have been designed around the 
restrictive assumption of a single working path between a source 
and a destination. The lack of native multipath support constrains 
network performance even as the underlying network is richly 
connected and has redundant multiple paths. Computer networks 
can exploit the power of multiplicity—through which a diverse 
collection of paths is resource pooled as a single resource—to 
unlock the inherent redundancy of the Internet. This opens up 
a new vista of opportunities promising increased throughput 
(through concurrent usage of multiple paths) and increased 
reliability and fault-tolerance (through the use of multiple paths 
in backup/ redundant arrangements). There are many emerging 
trends in networking that signify that the Internet’s future will 
be unmistakably multipath, including the use of multipath tech¬ 
nology in datacenter computing; multi-interface, multi-channel, 
and multi-antenna trends in wireless; ubiquity of mobile devices 
that are multi-homed with heterogeneous access networks; and 
the development and standardization of multipath transport 
protocols such as MP-TCP. 

The aim of this paper is to provide a comprehensive survey 
of the literature on network-layer multipath solutions. We will 
present a detailed investigation of two important design issues, 
namely the control plane problem of how to compute and select 
the routes, and the data plane problem of how to split the flow 
on the computed paths. The main contribution of this paper is a 
systematic articulation of the main design issues in network-layer 
multipath routing along with a broad-ranging survey of the vast 
literature on network-layer multipathing. We also highlight open 
issues and identify directions for future work. 

Index Terms —Multipath, Diversity, Routing, Load Balancing. 


I. Introduction 

It is common for communication networks to be built with 
significant redundancy for the purpose of fault tolerance and 
reliability. This results in the availability of multiple paths that 
can be used for connecting network nodes. These multiple 
paths can potentially be resource pooled for the purpose 
of routing and forwarding packets, and can be used either 
simultaneously or in a backup configuration, for increased 
reliability, end-to-end throughput, network efficiency, and fault 
tolerance. Unfortunately, despite the widespread path diversity 
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available on the Internet, traditional Internet technologies have 
focused mainly on single-path routing for the sake of its 
simplicity and lower overhead. This has led to the artificial 
lockup of Internet’s capacity, lower fault tolerance and reli¬ 
ability, and Inflexible support for quality of service (QoS). 
This is even in contrast to telephone networks, which have 
traditionally adopted multiple-path routing since it leads to 
better reliability and greater customer satisfaction (due to the 
lesser probability of call blocking) Q. 

Notwithstanding the lack of native multipath support on 
the current Internet, there is a growing convergence in the 
Internet community that multipath routing and forwarding, or 
mulripatliing in short, will play a big role in the future Internet 
by removing the artificial constraints of single-path routing. 
As we shall discuss next, single-path based approaches are 
deficient on many accounts. 

A. Deficiencies of Shortest Single-Path Routing 

Tliere are two fundamental drawbacks that traditional rout¬ 
ing techniques based on shortest single-path routing suffer. 

Firstly, it potentially limits the network throughput since 
only one path is accessible per source-destination pair. It has 
been shown that single-path routing is constraining optimal 
routing solutions, and that elegant optimal routing solutions 
can be developed if multiple paths with flexible splitting are 
utilized j^. The problem of joint optimization of routing 
and congestion control which belongs to the category of the 
general multi-commodity flow (MCF) problem is considered 
in Q for both single-path and multi-path configurations. It 
is shown that this problem is non-convex and NP-hard when 
the user can utilize a single path only. On the other hand, if 
all available paths can be used with source-based multipath 
routing, the problem is convex and admits an elegant solution. 
Wang et aJ. Q go on to investigate the performance loss of 
being restricted to a single path. This performance penalty 
of not using multiple paths is referred to as ‘the cost of not 
splitting* 0 0 This performance penalty, although hard to 
determine (being an NP-hard problem), is known to be strictly 
positive. 

Secondly, it is susceptible to routing oscillations since any 
changes in link lengths can cause abrupt traffic shifts. It 

'Traditional circuit-switched public switched telephone networks (PSTN) 
use a type of multipath routing known as hierarchical roiUi/ig or alternative 
path routing. Each exchange typically maintains precomputed routing tables 
with multiple alternative routes to any given destination with a fixed first- 
choice route, and one or more alternative routes hierarchically arranged for if 
the first-choice route is blocked, from which the exchange selects the route 
at the lime of the call depending on network congestion level Ql. 


is well-known in the literature that adaptive, shortest-path 
routing protocols—especially those that rely on link delays— 
can result in oscillatory behavior and instability leading to 
performance degradation 0 0 An example of an early 
del ay-based adaptive routing algorithm is the distance-vector 
ARPANET routing protocol jT) that was later found out to be 
unstable 0 0 

Optimal routing based on the rigorous mathematical the¬ 
ory of the MCF problem was proposed to eliminate several 
disadvantages of single-path routing by optimizing an aver¬ 
age delay-like performance measure Q. Opiimal routing is 
essentially a multipath routing framew ork. In optimal routing, 
traffic from a source-destination pair is split at strategic points 
allowing gradual shifting of traffic among alternative paths. 
Optimal routing aims to send traffic exclusively on paths that 
are shortest with respect to some link lengths computed on the 
basis of flows carried on those links. In optimal routing, for 
low input traffic, a source-destination pair tends to use only 
one path (which is the fastest in terms of packet transmission 
time), while for increasing input traffic, additional paths are 
utilized to avoid overloading the shortest path. 

B. Why the Internet's Future is Unmistakably Midiipath? 

Tliere are five main trends that indicate that the future In¬ 
ternet is going to be intimately intertwined with multipathing. 

7) Datacenters Trend: With the increasingly central role 
of datacenters (DCs) in the modern Internet, multipathing is 
becoming an explicitly designed feature of modern architec¬ 
ture. Unlike the traditional networking topologies that have 
evolved spontaneously, DCs are highly engineered architec¬ 
tural artifacts that have intrinsically incorporated multipathing 
for increased performance and reliability. 

2) Wireless Trend: In wireless communication, multipath 
plays an important role at various layers of the protocol 
stack (including the transport, network, and physical layers). 
Apart from the Importance of multipathing at the transport 
and network layers due to the multihoming of modem mobile 
devices (.such as smartphones), multipath also plays a critical 
role at the physical layer. Once considered as an antagonist 
of efficient wireless communications, due to the deleterious 
effects of multipath fading in which the reflected echoes of 
the transmission cancel each other destructively atfecting suc¬ 
cessful reception, multipath is now considered advantageous 
at the physical layer witli the recent advances in adaptive 
signal processing and antenna techniques in multiple input 
and multiple output (MIMO) technologies. To keep up witli 
the growing bandwidth requirements of modern applications, 
almost all of the emerging wireless technologies rely on 
the modem trend of equipping wireless nodes with MIMO 
antennas. With multiple antennas relying on the extra spatial 
degrees of freedom, the ability to successfully use multiple 
paths has become critical to the success of modem wireless 
networks. 

3) Multiple Wireless NICs Trend: Modern wireless network 
technologies have also made it feasible to equip network 
nodes with multiple network interface cards (NICs) tuned 


to orthogonal channels. The ready availability of commodity 
NICs, and the availability of modem networks to support 
multiple interfaces, has opened up the possibility of the 
successful exploitation of multipath technology to improve 
network performance. 

4} Muitihoming with Heterogeneous Technologies: Muki- 
homing is a technique that has been traditionally used by 
server farms on the Internet for reliability and high per¬ 
formance. Analy.sis of muitihoming has shown significant 
performance benefits that accnie with multhihoming, with a 
careful selection of three or more Internet service providers 
(ISPs) providing pertbnnance improvement, in terms of trans¬ 
fer throughput and round-trip-time (RTT), of upto 25% [ [TI?) . 
While traditionally muitihoming has been relevant only for 
critical servers, it has now become a mainstream with billions 
of mobile devices now multihomed with heterogeneous access 
technologies (such as Wi-Fi/ 3G). A wireless device (such 
as a smartphone) can exploit its interfaces to heterogeneous 
technologies in order to improve its reliability, throughput, 
and fault tolerance. In recent times, there have been multipath 
transport-layer initiatives that are built specifically to incor¬ 
porate such muitihoming. In particular, the .standardization of 
MP-TCP pT) in RFC6824 has led to mainstream applications 
of multipath: e.g., the Siri intelligent personal assistant on Ap¬ 
ple's iOS7 uses MP-TCP to increase efficiency and reliability. 

5) Path Diversity in ISPs: Teixeira et al. have studied 
the availability of path diversity (the availability of multiple 

g iible paths between two communicatina entities) in ISPs 
. It was shown that a typical large ISFp has a significant 
path diversity. The abundant availability of path diversity does 
not necessarily imply that the best paths are adopted though— 
in fact, statistics show that at any lime 30 to 80 percent of the 
Internet traffic chooses a single suboptimal path even though 
alternative low-loss lesser-delay paths exist (T4) . With the 
inherent path diversity in ISPs, there is a strong interest in the 
service provider community to exploit multiple paths fTTj . 

C. Benefits of Multipathing 

We provide a description of some of the major applications 
and features of multipathing below: 

I) Resource Pooling: Resource pooling is a broad In¬ 
ternet architectural design principle—focusing on improving 
resource efficiency by presenting a collection of resources 
as a single pooled resource—that has many manifestations. 
For example, some of the most disruptive innovations on the 
Internet—packet switching, content-delivery networks (CDN), 
peer-to-peer networks (P2P), cloud computing—are based 
on resource pooling p6) . The packet switching technology, 
which acts as a foundation of the Internet, is essentially 
a resource pooling technique. To put things in perspective, 
circuit switching techniques use fixed multiplexing schemes in 
which dedicated and isolated non-pooled circuits are utilized. 
In contrast, statistical multiplexing—the underlying technique 

"Empirical caJculalions were performed on a lier-I Sprint ISP network using 
heuristics for inferring path diversity based on active measurements using the 
Rockelfuel p3) toolset. 


of packet switching—achieves resource pooling by allowing 
a burst of traffic on a single circuit to use spare capacity 
on other circuits. CDNs and P2P networks utilize resource 
pooling to ensure efficient network operations. P2P protocols 
such as BitTorrent use swarming downloads that let receivers 
pool together multiple peers as a data source thereby using 
multipath to pool the network paths to these peers. Cloud 
computing relies on pooled resources gathered at central DCs 
to support the computing requirements of multiple tenants 
through virtualization. Multipathing has been envisioned as 
'Pocket Switching 2.0' [ [T5| since it allows an analogous 
resource pooling benefit at the network layer. In particular, 
multipath routing can pool together a number of distinct links 
to provide the abstraction of a single unified network resource. 
With the abstraction of a single resource pool, the chances of 
a demand being refused due to one’s resources being utilized 
reduces. 

2) Load Balancing: With resource pooling, a collection of 
resources acts as a single virtual resource. The biggest promise 
of resource pooling is load balancing that allows congestion 
control techniques for diffusing congestion over resources by 
equitably di.stributing the load amongst the elements of a 
resource pool. Thi.s help.s to avoid situations where a certain 
network resource unnecessarily acts as a bottleneck although 
the network has alternative resources that can help to relieve 
the congestion (TT) , p^ . Load balancing can be done over 
resources (e.g., the load balancing over multiple access links 
and paths as done in multihoming and multipath transport 
respectively) or can be temporal (i.e., the load is balanced 
between peak and non-peak hours to exploit the typical diurnal 
network traffic pattern). Load balancing over time shifts the 
transmission of data from peak time to off-peak time and 
can help in reducing costs such as the commonly used 95 
percentile pricing scheme eg. 

3) Efficiency: Another major benefit of resoure pooling 
through multiple paths is efficiency. It Ls well known in the 
literature that the Internet traffic is bursty with a pronounced 
correlation with a large variance of the traffic volume over 
long time periods m ED Allocating fixed amount of 
resources to individual circuits is inefficient due to the bursty 
nature of the Internet traffic, which can result in largely idle 
resources or resources that are insufficiently provisioned. One 
important benefit associated with resource pooling is that it 
leads to increased efficiency. Resource pooling, or .statistical 
multiplexing of shared resources, suits the Internet traffic 
well. Thi.s is because the law of averaging (or the law of 
large numbers) does not apply to individual flows. Resource 
pooling is useful since it exploits the statistical regularity of the 
aggregate of individual bursty circuits. This results in efficient 
utilization of resources without extensive over-provisioning. 

4) Reliability and Fault Tolerance: With the migration of 
critical businesses to the Internet, the reliability and avail¬ 
ability of the Internet has become extremely important. Fault 
tolerance mechanisms for networks without multipath support 
are unwieldy with a link/ node failure potentially resulting 
in significant recovery times. The resource pooling benefit 
of multipathing allows the underlying network to gracefully 


handle loss of capacity or the failure of individual link.s/ 
paths through diffusing thi.s anomaly by shifting traffic to the 
working members of the resource pool. This allows enhanced 
fault tolerance and helps to build more resilient networks. 

Network communication can utilize multipathing to enhance 
its reliability at various layers |22| . At the network layer, 
we can have multihomed servers that have different ISP 
connections; another option is to utilize backup paths with 
traffic being switched to an alternative path in the case of 
failure of the primary path. At the physical layer, techniques 
such as antenna diversity can be used to combat fading effects 
in wireless communication. 

Multipathing can also be used to facilitate network security. 
More specifically, the use of multiple radio interfaces has 
been proposed for achieving greater data confidentiality by 
encrypting data and splitting it into multiple parts that are 
then transmitted using different physical channels (3^. Mul¬ 
tipathing can also be used to mitigate malicious adversaries 
that drop packets intentionally on a path by circumventing the 
adversary using an alternative disjoint path if available. 

5) Higher Throughput: A great number of modem high- 
performance Internet technologies utilize multipathing in one 
way or another. At the physical layer, MIMO technology 
is revolutionizing the wireless industry by promising sig¬ 
nificantly enhanced communication rates. The technique of 
space-division multiple access (SDMA), u.sed by multi-u.ser 
MIMO (MU-MIMO), allows multiple transmitters to send— 
and multiple receivers to receive—separate signals simultane¬ 
ously over the same band. The improvement offered through 
multipathing at the link layer and the network layer is dramatic 
in environments where the traffic is concentrated on only a 
few links while other perfectly fine links are not used at all 
(e.g., in bridged networks where the spanning tree protocol is 
used). Other high-performance proposals include mullipath- 
TCP (MP-TCP) at the transport layer that allows the pooling 
of multiple links into a pooled connection to support higher 
speed: for example, there has been a recent demonstration of a 
transfer achieving a goodput of 50 Gbps using MP-TCP with 
nodes equipped with 6-10 Gbps commodity NICs j24| . 

6) Flexibility due to Pluralism and Multiplicity: In re¬ 
cent literature [ ^5) , the metaphors of “the hedgehog and the 
fox'[^]— popularized by Isiah Berlin in an essay of the same 
name—are being used to highlight the importance of flexibility 
that accrues with pluralism. For example, it has been shown 
that a forecaster who systematically entertains a variety of 
possibilities outperforms a rigid one-dimensional forecaster 
fixated on a single potential interpretation pS) . There is also 
a wealth of literature that supports the claim of "the wisdom 
of crowds" [ |27| . 

Many of the benefits of multipathing are brought about 
by the 'diversities' it offers. The benefits of diversity have 
been established in a broad .spectrum of fields ranging from 
communications, biology, economics, to sociology. Intuitively 
speaking, diversity is useful in a variety of fields since it allows 
hedging in uncertain situations to minimize risk and thereby 

Based on a reporled saying from Ihe ancient Greek poet Archilochus who 
hud said: "ihefiix knows inanv things but the hedgehog knows one big thing". 
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downplays the negative effects of uncertainty. 

7) Support for Multimedia Applications: Tlie modem rise 
of high bandwidth multimedia applications can benefit signif¬ 
icantly from the resilience offered by multipath techniques. 
While the traditional packet-switched Internet is inherently 
a best-effort network where a single path is mostly in use, 
multiple paths can coexist at the same time. As scarce 
bandwidth and packet loss are more significant problems for 
today’s streaming media because of its increased transmission 
requirements, concurrent transmission of multimedia appli¬ 
cations over multiple paths can be adopted to resolve the 
aforementioned problems. 

8) Role in the Future Internet Architectures: It is antici¬ 

pated that multipathing will play a major role in the future 
Internet architectures and next-generation wireless and mo¬ 
bile networks. It is already typical for mobile phones to be 
multihomed with heterogeneous connectivity (e.g., Wi-Fi and 
3G). The upcoming 5G standard (expected to come to fruition 
around 2020) is expected to integrate and resource pool a 
number of diverse access technologies (25) with multipathing 
playing its part in seamless handovers and interworking of 
heterogeneous technologies. With the resource pooling of 
cellular and Wi-Fi networks, the possibilities of Wi-Fi offload¬ 
ing and cellular unloading emerge. With Wi-Fi offloading, 
the performance and scalability of mobile networks can be 
improved by offloading data traffic to the Wi-Fi networks [ [19) . 
With cellular onloading, on the other hand, home broadband/ 
Wi-Fi connections are augmented with cellular data services 
for bandwidth aggregation j^O) . Apart from wireless networks, 
multipathing is also useful in emerging networking configura¬ 
tions (such as DCs and clouds) and in other upcoming Internet 
architectures such as P2P. CDN, software-defined network 
(SDN), and content-centric network (CCN) (??) . 

9} Context-aware Services: It should be noted that different 
networking applications have different needs. In particular, 
various application configurations can coexist: e.g., the first 
application prefers a low-delay path, the .second prefers a 
high-throughput connection, and another prefers a reliable 
connection. Using multipathing, context-aware .services are in 
line with the application requirements. 


D. Challenges in Implementing Multipath Routing 

Even though the future of the Internet promises to be inter¬ 
twined with multipath routing, there are numerous challenges 
that need to be surmounted. A major challenge in implement¬ 
ing Internet-wide multipath routing is to achieve scalability 
due to the resulting considerable computational and storage 
overhead. Specifically, multipath routing entails significant 
overhead both in the control as well as the data plane. Control 
plane overhead refers to the need of additional bandwidth 
and processing resources for exchanging path infonnation, 
while the data plane overhead refers to the large forwarding 
table and the higher memory requirement associated with 
forwarding entries corresponding to an increased number of 
paths. Another stumbling block is the lack of complete control 
of any organizations over tlie entire end-to-end path. We will 


discuss more open and current research issues later in the paper 
in section [13 


E. Organization and Contributions of This Paper 

We organize this paper around five fundamental design 
issues that relate to network-layer multipathing. These designs 
issues are presented below in the form of design questions: 
Ql. Route computation: How to compute the multiple paths 
to be used? 

Q2. Route metric: What routing metrics should be u.sed to 
construct the ‘best’ multiple paths? 

Q3. Load balancing/ Flow splitting: How to have a traffic 
flow use the constructed paths? 

Q4. Number of paths: How many paths should be constructed 
and used? 

Q5. Concurrent usage: Should the multiple paths be used 
concurrently or as backups? 

Tlie overall organization of this paper is illustrated in Figure 
Tlie main questions that we analyze, and present solutions 
for, have already been presented. In Section we discuss 
various techniques that can be used for the construction and 
selection of multiple paths (Ql) and also the metrics that are 
used for facilitating route construction (Q2}. In Section 
we tackle the issue of how to use the multiple constmcted 
paths (Q3). In particular, we discuss issues relating to load 
balancing, flow splitting, and multipath congestion control. We 
discuss the choice of how many paths to use (Q4) in Section 


Ill-B and whether to adopt backup or concurrent multipathing 


(Q5) in Section Ill-Bj In Section [Iv| we present potential open 
research issues in the field of multipathing at the network layer 
followed by our main conclusions in Section [V| 

The main contribution of this work is a comprehensive 
survey of network-layer multipath routing in which apart 
from our network-layer focus we also provide a balanced 
and holistic cross-layer coverage of multipathing (recognizing 
the fact that many important issues in multipathing—such as 
resource pooling, congestion control, fiow splitting—involve 
cross-layer interaction. While there exist quite a few survey 
papers on multipathing protocols, these papers are typically 
confined to particular networking configurations (such as wire¬ 
less sensor networks 034| , (5^, PS) , mobile ad hoc networks 
( PT) , pS) , P9) , etc.) We will present a more general paper 
that will comprehensively describe all the important multipath 
proposals/ ideas and will also tie them up with new advances in 
transport-layer multipathing. With the increasing importance 
of multipathing on the Internet, there is a strong need to 
have such a paper to benefit practitioners and researchers in 
the networking community. This paper is timely due to the 
huge interest in this domain (emerging from new research/ 
standardization work on multipath protocols, such as MP- 
TCP). The success of higher-layer multipath protocols depends 
critically on support from lower-layer protocols (in particular, 
the network-layer protocol). 


II. How TO Construct Routes? 

On the Internet, the task of the construction of routes is 
handled by the control plane, while the task of forwarding 
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Fig. 1: Various network-layer multipathing design choices (and their coverage in this paper). 


the packets based on the computed routes is performed by the 
Internet's data plane. Tlie Internet's control plane is typically 
managed through dynamic routing protocols. Generally speak¬ 
ing, it is desirable for a routing protocol to (i) choose ‘optimal' 
paths, (ii) be robust to dynamic network topology in order 
to avoid instability in the form of oscillatory route flaps, and 
(iii) minimize routing overheads in terms of memory required, 
messages exchanged, etc. There are several choices in the 
design of multipath routing protocols—including centralized 
or distributed routing, static or dynamic routing, source-based 
or hop-by-hop routing—corresponding lo different tradeoffs 
with respect to complexity and resource usage. 

In this section, we focus on the various techniques pro¬ 
posed in the literature for the computation of routes. These 
techniques include “source routing" in which the end-to-end 
route is computed and specified by a source and “hop-by-hop 
routing" in which network routers exchange control packets 
to establish state that is then used for hop-by-hop forwarding. 
Broadly speaking, the various multipath routing techniques 
can be classified into four categories of: (i) source routing, 
(ii) hop-by-hop routing, (iii) overlay routing, and (iv) SDN- 
based routing. In the following subsections, we discuss each 
of these categories in tliis order. A summary of the multipath 
proposals presented in this section—along with a description 
of their route computation technique, routing metrics adopted, 
and specific networking configuration assumed—is presented 
in Tables [I] [IT] and [III} 

A. Source Routing 

In many cases, the network is not well placed to know about 
the optimality of a route. Tlie source host/ user, on the other 
hand, knows about tlie application's unique requirements and 
can judge about the appropriateness of a route. Tlie source is 
also best suited lo deal with any failures that may arise along 
a route. This motivates source routing in which the source 


host, or edge router, defines the end-to-end path to be used 
for forwarding packets with the potential to improve network 
performance, reliability and user satisfaction. 

In the source routing paradigm, the source explicitly embeds 
information in a data packet about the path that the packet 
will traverse before reaching its destination |47| . Such a 
scheme proceeds typically by first discovering routes through 
the exchange of control messages and then embedding the 
discovered route information in the data packet's header. The 
routers along the explicitly defined path then forward the data 
packet using the information stored in the data packet's header. 
Although source routing provides maximum flexibility, it is 
important lo point out some difficult challenges that it faces 
[ [97| including (i) limited control of intermediate autonomous 
system (AS); (ii) scalability problems due to the need of 
having the source to compute and define routes; and (iii) 
efficiency and stability problems that may arise from selfish 
source routing. 

In the next two subsections, we will describe source routing 
proposals, initially discussing general solutions (developed 
mainly for wired networks) in Section |II-A1[ and then dis¬ 
cussing source routed multipath solutions specific to wireless 
networking in Section |1I-A2| 

/) Source Routed Miiltipailiing (General/ Wired Solutions): 
To accompany this section, we present a comprehensive listing 
of general multipathing solutions (designed primarily for wired 
networks) in Table [I^ This table serves as a convenient 
summary to complement the description of various hop-by- 
hop multipating routing solutions in this section. In addition, 
the table also includes information about works that have used 
other routing schemes (source routing, overlay routing, and 
SDN-based routing), and we will be revisiting this table later in 
this paper. We note here that the classification is approximate 
since not all works can be cleanly pigeonholed into only one 
of our broadly defined categories. 
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TABLE I: Frameworks for Determining Multiple Paths (Route Selection) 

Proftd Km/* Deieri/fHoii 


Soared Rotuing 

Split MulUpoih Routing (SMRl poj. 

Multipath MPLS p2] '—* ^ 

Feedback based nTOTSig iFBRi [43| 

BANANAS Q 

Single Hup Source Rouuag <SOSR> 

spread!^ 

Routing DeSection>i 

)(Ani Mulu-chunnel Mu) ti-puth Control iJMM) 

New Inimei Routing Archiieiiture {NIRAi 
Path Splicing |S0j 
PutKIel Routing 
Slick PBCkeln [52| 

AMrR|S3| 

Hop-by-Hop Reuliag 
Ksper»iity routing |s^ 

Geographical Power E6icient Routing (GPERl 

Loop-free Invariant (LHi-based Approaches |^. |^. H 

HuHhing-based ECMP 

Snnivus and Modim w 160 1 
Puckei-caching |^, jfcp 

Interference Mlnituizaiion Appruacheit (M-MPR d- El- 
Muituuurl^) 

AODV-enMed Routing Schemek IAOMDV |^; Ye ei ed 
MP-ADDV INDMR > 


1999.2000 
2001 
2002 

2002 

20M 

20M 

2006 

2007 

2007 

2008 
2009 
2011 
2012 

1975 

20M 

1999.2001 
2000 
2002 

2002,2010 

2002. 2008 

2001.2003. 

2004.2004 


Biolugicaily Inspired Routing 1^, [tT] 

Securitv-Eftfuin Led Ao pruaches inbir. aMBR. H-SPREAD) 
Enhanced BGP|7^.|^ 

MuKitsum Alternative Roudng AJgonihiu IMARA) 

Huang and Fung j75j 

Network coding based reliable disjoint and braided oiuliipaili 
routing protocol (751 

Yet another mulim3Qi routing (YAMR> 1771 
Behagy ei al. (tI) '—* 

Owbty Routing 

Deroule 0 

Resilient Overlay Neiwoi4t IRON) 

Multipath Video Streaming on Overlay Networks 

Multipath multiple description video nretiming [bi] 

BARON '—' 


2005 

2006.2011 
2007.2009 

2009 

2009 

2010 

2010 

2011 


1999 

2001 

2004 

2005 
2008 


Applies a probing-based mechanism to achieve a balanced load over multiple constructed path.^;. 

Constructs multiple paths in polynomial time subject to a constraint (i.e.. number of hops and paths). 

Performs routing decision at the access router with two routes computed for each network prefix using dynamic quality 
momioring based on suttctural infnrmaiion disseminated by the iranxli routers . 

Enables multipath routing in conventional routing protocols (e.g., OSFP and BCP) using global identifiers for 
abstraction 

Uses randomly chosen iniermediaries to route packets indirectly towards the eventual destinations in order to recover 
from faJures which occur near the destinations. 

Divides a secret message into multiple pans and sends them over muliiple independent paths towards tbe desiinaiion in 
order to improve »ecrecy and security. 

Provides path diversity by allowing routers to redirect packets based on tagged informauon provided by end hosts, 
subject to the compliant of the ISP routing policy. 

Achieves a balanced distribution of the contending input traffic at different limes over dllTereni channels and difTereni 
paths through cross-layer Interaction between link and network layers. 

Provides path diversity by enabling a packet in a source controlled mannet to switch among multiple ISPs enrouie to 
iu destinaiion. 

Constructs a path by c^ombining slices of multiple routing trees, which serve as backups paths, en route to an eventual 
destination to improve reliubdiiy. 

Performs rouimg deflection ai the domain level of a hierarchical network. Each domain advertises a few paths or paibfeis 
used by the a source from another dotuain. 

Embeds alternate route Information is embedded in the packet's header in a source controlled fashion to avoid failures. 
This sdieme requires minimal network routing stole because of the source routing feature. 

Establishes multiple pailis around the primary paths in hierarciiical networks. 


Sends distinct data packets over multiple paths for load balancing, or sendx duplicated data packets over multiple paths 
for improved leliabiLiy and resiliency. 

Uses probabilistic multipailnng to achieve patb diversity The nodes are rendered to choose their transmission ranges to 
conserve energy. 

Incorporates multipath routing in undiiional link state-based and di stan ce vector-based aJgontbms to fmd loop-free 
multiple paih.s using LFI condition Diffusing compuiurivn Le Used in (551. 

Selects patlis from a set of etfuat cost muliipU paths in order to spliHlows for load balancing. It comprises of direct 
hashing and table-based hashing methods. 

Uses 2-link-disjoini and k-noJe-disjoini algorithms to conserve energy. 

Cirvumvenis route breakdowns by caching/ storing (a fixed number off data packets along with alternate route 
information at nodes in a distributive fashion 

Constructs multiple paths in an incremeniaJ fashion keeping in view the congestion and interference effects (using the 
inierpath interference and disjomtness metrics) M-MPR incorporates FEC and selective forwarding. 

AOMDV esiohluthes multiple link-disjmni paths in an on-demorvd manner: Ye ei al. use reliable nodes, in terms of 
energy resources, to establish multiple paths: MP-AODV selecb primary paths u.xing received signal strength, and 
alternative paths tlirvugli heartbeat message exchongeh: NDMR incorporaieh mutiputhing iniu traditional AODV and 
DSR. and uses alternative paths as backups 

Uses b^logicolly inspired approuchek (e.g.. oiii colony optimization) to establish and maintain multiple paths 
Enables nodes to take independent decisions to address joint fault and iniiuxion tolerance. 

Extends the traditional BCP routing protocol to esiablish and maintain loop-free multiple paths in the fac'e of multiple 
failures in hierarchical networks. 

Constructs directed acycUo graphs (DAGs) that (i) maximize the min. connectivity: dil maximise the min max. fiows: 
and (iii) maximize the min max. fiowk us on exiension of shonesi-path rouimg. 

Selects multiple parent nodes in an opportunistic manner to establish multiple paths even though nodes may have 
unknowo speeds in vehicular ad-hoc netwotks. 

Uses braided and disjoint multiple paths in a distributed fashion in conjunction with network civding for relialnlity and 
energy conservation purposes. 

Establishes multiple paths, and providex concurrent transmission with minimal overhead m hierarchical networks. 

Uses multiple paths to reduce negative effects from and to licensed users in cognitive radio networks. 


Proposes an edge-based routing scheme for the Detour overlay network comprising geographically distributed routes 
fur better end-to-end performance 

Focuses Oft equipping end hosts and applications with the abdliy to kwlich between direct routing (through the 
underlying network) or indirect routing (using the overlay network). 

Proposes a link correlation-bused QoS metne along with path ourrelation model for muliipaihing using on overlay 
network fur efficient video sirearaing. 

Selects d set of multiple paths bused on multipit Jtu'nprion lading modeK. 

Proposes a bondwidth-aware routing scheme for overlay networks to meet application bandwidth requirements and 
improve the overall throughput performance. 


SD.y-Bastd 

Hedera 

MicTO-TE 

Rain Man 1851 
B4 iGooglh^ 


Wireless Mesh Software-defined Networks (wmSDN) |87| 
Software Defined VANETs IfiSl ^ 


2010 

2011 


2012 

2014 


Proposes on efficient, scalable, adaptive and centra] scheduling method, with a universal scope of active fiowx in view, 
for multistage switching processes in datacenters. 

Implements central controlkrs using OpenFlow. ox well as uses short-term traffic predictability and multipath routing, 
to schedule Sows in a datacenter environment 

Proposes a layer two (L2 1 multipath packet forwardmg algonthm for a SDN-based firmware architecture for datacenters. 
Proposes a centralized TE solution that uullzes OpenFlow/ SDN principles along with multipath forwarding to Improve 
network efficiency (i.e.. link utilization and average network utilization). 

Achieves traffic balancing and selects gateways in on intelligent round robin fashion for Internet connectivity. 
Exiablisbes multiple paths lo address route failures, as well as for the transmission of high priotiiy fiows. in VANETs. 


Due to the lack of flexible support for traffic engineering in 
IP-based networks, the multiprotocol label switching (MPLS), 
with its rich support for traffic engineering (TE), has become 
ubiquitously deployed in ISPs. A lot of work has been done on 
etfleient optimized multipath TE. Lee et aJ. p2I| proposed a 


constrained multipath TE scheme for MPLS-based networks to 
help find optimal paths that meet the provided traffic demand 
as well as the constraints (such as maximum hop count and 
node/ link preferences). Tlie proposed scheme calculated the 
flow split ratio for the multiple paths. The evaluation of 
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TABLE II: Summary of General Multipath Routing Proposals (Designed Mainly For Wired NeTMorks) 


Wwk 

Year 

Rauir ConKiruclIon 

Palhi 

,%fetriiit) 

Dncripllon 

Dispersiiy Rouiiny 

1975 

Hop-by-Hop 

Concurreni 

Delay, ReliubUiiy, Loud 
BdJuncing, Link Uiiluaiion 

Disperses irolGc over nultiple ptnhs either in a ruJunJani or non-redurvdani manner 

OSPF-OMP [^. 

1999 

Hop-by-Hop 

Concurreni 

Delay, Loud Buloncing 

Provides a hashing-bused OSPF extension lo support upiiniioed mulupaihing for load 
buloncnng. 

ecmp[?T] 

2000 

Hop-by-Hop 

Concurreni 

Delay, Loud Buloncing 

Rjrwords packei:< based on a CRCI6 bash lo one of the nodes fonaing equal-cosi paihs 
iwards u desiinauon 

MuiapaiK MPLS[^ 

200) 

Source RouUng 

Concurreni 

UnkLiUliidiion, 

Load Balancing 

Cuosimc*ts multiple paOcs in polynomial lime subject to a consiroioi (i.e. number of hops 
and paihsi. 

FeeUbjck-bii»ed Routing 

iFRRlp^ 

2003 

Source RouUng 

Buckup 

Delay 

Enables on access router to cumpuie two routes for each network pre6i based on the 
siniciuroJ infurmoiion disseminated by ininsii routers. 

BANANAS |44| 

2003 

Source RouUng 

Concurreni 

RHuurce UiJuaiion 

Provides a framework that creoles iffohal tileniifier to enable multipathing in the (niemeL 

SOSR [4S| 

2004 

Source RouUng 

Buckup 

Overhead. ReliabJlty 

Presents a single hop soune routing (SOSR} to avoid failures which occur near the 
desiinaiions in the Intemei 

MONET |92| 

2005 

Overlay Routing 

Buckup 

Overhead. Delay, 

Reliabiliry 

Provides multiple backup paths through proxies lo websites for enhanced web availaUlity. 

Rouuag De/leciioni 

2006 

Source Rouung 

Concurreni 

Overhead, Reliability 

Provides an (SP that construct altemoie paths or based on the fug.x thai users put 

in their pockets. 

MIRO 

2006 

Inicrdooiain Nego- 
uairoas 

Backup 

ReliabilUy 

Provides interdomoio negotiations while esiablishmg paths by improving upon BGP and 
source routing. 

R-BGP [tII 

2007 

Hop-by-Hop 

Backup 

ReliabilUy 

Provides alternate paibx to overcome transient failures by modifying BGP. 

NIRAp^r 

2007 

Source Ruuung 

Backup 

Reliabiliiy, Overhead 

Enables a user lo choose the sequence of ISP« for Us packets m order to switch among 
multiple routes. 

Pjih Splicing 

200S 

Source Rouung 

Backup 

ReliabilUy 

Allows likes of different routing trees to be spliced together for reliable delivery of a packet 
towards a desiioalion 

D-BGP and B BCP 

2009 

Hop-by-Hop 

Backup 

Overhead, Reliability 

Provides a path diversity-aware routing protocols in which an advenisemeni method is 
proposed for multiple path.^;. 

Poihlei RouLog |5^ 

2009 

Source Rouung 

Backup 

ReliabilUy 

Allows a source to concaienaie and adverlise segments of paibs or paihleis in packets while 
coosinjctmg oo end-io-eod path. 

MulUp4ih BGPp3| 

2009 

Hop-by-Hop 

Concurreni 

Resource UiJizaiion, 

Uses multiple paths in a concurrent fashion and preserves loop-freeness by modifying BGP 

YAMR |77| 

MulU QaDrEGP 

2010 

Hop-by-Hop 

Backup 

ReliabilUy, Overhead 

Coosuvcts a set of multiple path.^; to avoid lat least) ao inierdonuun link failure. 

2011 

Hop-by-Hop 

Concurreni 

Resource Uidizaiion 

Proposes two muliipathing appntaches at the inierdimiain level. 

AMIR 1?^ 

2012 

Source Rouung 

Concurreni 

Reliability 

Coosuvcis alteroaiive paths around the pnmaiy path using topology information from 
oeighboriog ASes. 


the proposed scheme showed superior results in utilization, 
traffic volume, and overhead compared to the conventional 
shortest path routing scheme. In another work, Seok et al. 
^7\ proposed to integrate MPLS traffic engineering with 
source multipath routing subject to a constraint on the total 
number of hops and the total number of paths. Tlie presented 
algorithm calculated the flow split ratio by using hashing 
over the calculated multiple paths as per the traffic demands 
dictated by an ingress-egress pair of a network. A dynamic 
traffic engineering approach is proposed to tackle the issue of 
congestion as the heuristic presented in this work protects a 
link from the overwhelming effect of huge traffic demands by 
minimizing the maximum utilization of a link and using the 
aspect of multipathing for load balancing. 

Approaches to Avoid Failure/ Performance Bottlenecks 

It was shown in that uplo 80% of the default routing 
paths on the Internet are suboptimal with an alternative route 
offering potentially lower loss rate. This has motivated the 
development of numerous source-routed routing protocols that 
can help route around the performance bottleneck/ failed 
entity. 

Gummadi et al. (45) proposed a source routing-based 
approach named "scalable one-hop source routing" (SOSR) 
to recover from Internet path failures by routing indirectly 
through randomly chosen intermediaries. It has been shown in 
the previous work that last-hop and end-host failures account 
for a disproportionately high number of path failures. Such 
failures are especially hard to recover as there is a limited 
number of network elements to route tlirough as the route 
nears the destination. SOSR recovers from such failures by 
routing packets indirectly using a small set of randomly chosen 
network intermediaries. When a network failure occurs, then 
the source probes the destination through the randomly chosen 


set of intermediary nodes. If such a probe is successful, 
then the data is routed via this indirect route; otherwise, 
it is concluded that the destination is now unreachable. In 
this technique, since the choice of the intermediary router 
is made randomly from a preselected set. the continuous 
maintenance of backup routes is avoided. Tliis allows SOSR 
to outperform other overlay or backup path techniques that 
require continuous path maintenance and are more expensive 
in terms of control overhead. 

Using the concept of path splicing (50) — which can be ap¬ 
plied in both interdomain and intradomain routing scenarios— 
a packet can traverse across segments or slices of different 
routing trees (based on the discretion of the end hosts). In 
path splicing, multiple routing trees are established with a 
packet able to switch from one routing tree to another at any 
hop/ edge. This concept has been shown to improve network 
resilience. 

Interdomain Source-Routed Multipath Routing 

To manage a global-scale routing, the approach adopted on 
the Internet is to separate routing into intradomain routing and 
interdomain routing, where the former focuses on “optimal” 
routing (defined in the terms of some optimization metrics) 
within a single AS, while the latter focuses on ensuring loop- 
free policy compliant routing between multiple ASes. Since 
most traffic on the Internet traverses across multiple domains, 
interdomain multi pathing is of great practical importance. 
Tlie defacto interdomain routing protocol of the Internet is 
Border Gateway Protocol (BGP). BGP allows different ASes 
to exchange reachability information that is then used by BGP 
for computing loop-free policy-compliant routing. 

Although BGP is well entrenched in the Internet ecosystem, 
it is widely known that most routes chosen by BGP are not 
optimal in terms of performance, cost, or reliability. There 





TABLE ill: Summary of Multipath Routing Proposals for Wireless Networks 




Year Route Paths MefHc(si 

CensirucHon 


CUiciipiion 


Wtrelesx Senior Nepswks iWSNsi 


Disjoint and Braided 

2001 

Hop-by-Hop 

Backup 

Energy. f^liabiJity. 
Overhead 

Proposes a reliable, ouo-disjoini or braided mabtpitthmg scheme that performs SQ% better 
than dujoini mulupaihing. 

Selective Forwarding 1631.164| 
MMSPEED[y7] '—* ' — * 

2003 

Hop-by-Hup 

Cuncurreni 

Enert'v. Thruuvhout 

Proposes a meshed multipath routing proiucol with Helective forwarding. 

2006 

Overlay Routing 

Cuncurreni 

Delay. Reliability 

Proposes a probabilutic multipath aelecilon wheme under reliability and QoS conxiroinu 
(i,e., delay). 

H-SPREAD 

2006 

Source Routing 

Cuncurreni 

Reliability 

Proposes a hybrid muliiputh method where node/ bnk-dlsjuint path.* ore maintained on each 
sensor nude. 

ebmr[^ 

2006 

Deterministic 
(Defined by BS) 

Backup 

Energy 

Enable* a BS lu discover, maintain and switch among multiple paths keeping in view the 
nodal residual energy. 

Totally Disjoint Paths |I00| 

2006 

Hop-by-Hup 

Cuncurreni 

Interference. 

Throughpui 

Proposes and studies a 2-paih routing scheme m a suigle and a multiple suurce-desiinaiion 
pair Hcenonus. 

GPERp^ 

2007 

Hop-by-Hup 

Backup 

Energy 

Enable* nodes to decide their iransmission ranges distnbutively fur power efficiency using 
probabJistic muliipathing. 

dmpr[To2| 

2007 

Hop-by-Hup 

Cuncurreni 

Load Balancing, 
Throughpui 

Cucstiucu two (colored) trees, which ore rooted al (he sink node. Two paths ore formed 
from a source via each tree, which is link/ nude disjoint 

SEEM|I03| 

2007 

Deterministic 
(Defined by BS) 

Backup 

Energy. Throughput. 
Overhead. Reliability 

Enable* BS to dlicover and miuntoin multiple routes, a.* well as switch among them, in 
order lu conserve energy and enhance Hecurify. 

Mullicosl Multipath jl04| 

2007 

Hop-by-Hup 

Cuncurreni 

Delay. Bandwidth 

Proposes three (ree-bosed on-denianJ and disinbuied multipath routing methods under 
multicast QuS constraints. 

Mmg et al, |lOs| 

2007 

Hop-by-Hup 

Cuncurreni 

Energy. Delay, Overhead 

Present a scalable, distnbuied. reliable and node-iUsjoini multipath routing proiocul to 
achieve efTlcieni load balancing 

Maximally Radio Disjoin! 
Routing 

Mullicorullnned QoS Rouung 

2008 

Hup-by-Hop 

CoDcurrenJ 

Energy, BondwidUi, lo- 
lerference 

Cooaifucts multiple paths in on incremental way lo reduce interference m multimedia 
WSNs 

2008 

Hop-by-Hop 

CoDcurrenJ 

Delay. E^liabilUy 

Cooairucts multiple paths based on a prubabibstic link-siaie method under delay and 
reliabiliiy QoS consiroinia. 

2009 

Hup-by-Hop 

Buth 

Energy, Delay. Reliabil- 
iiy 

Usei backup and concurrent muliipathing approaches to cooierve energy and enhance 
robustness. 

nc-rmr 

2010 

Hop-by-Hop 

CoDcurmu 

Energy, Reliability 

Uses both disjomi and braided multiple paths, with network coding, in a per-hop faobiun 
for energy conservaiion and reliability enhancement. 

sejf[7^ 

2011 

Hup-by-Hop 

CoDcurrem 

Energy, Reliability 

Proposei a distributed. BS independent and one-way hoah chains for muliipoihing 


2014 

Hop-by-Hop 

Coocurreni 

Energy, Reliability 

Preaeoi^i an iniefeience-mimmized multipadi routing protocol (IM2RP^ that aims to dis¬ 
cover minimum interfering paths 


Mobile AO-hae Setmorks (MASETsi 


On-demaAd |40| 

1999 

Source Routing 

MSR|41| 

2000 

Source Routing 

smr [ttd| 

2001 

Source Routing 

AOMDV[S| 

2001 

Hup-by-Hop 

mp-dsr(TTT1 

2001 

Source Routing 

Yeeial. 

2003 

Hup-by-Hop 

CHAMP 1^ 

2003 

Hop-by-Hop 

Mm. Energy Paths 

2003 

Hop-by-Hop 

SPREAD p6| 

20M 

Source Routing 

Serial MuTD^ Disjoint Trees 

20M 

Source Routing 

Muliicati Routine r Serial 



MriTMR 1 rrn 



mp-aodv |u| 

20M 

Hop-by-Hop 

NDMR |69|^— 

20M 

Hop-by-Hop 

Clienei al. jlli| 

200n 

Hop-by-Hop 

MnltinntK 01 fTu] 

2005 

Hop-by-Hop 

AnlHocNei[70|.[7T|"' 

2005 

Hop-by-Hup 


Backup Overhead 

CunLurreni LuaJ Balancing 
CuHLurreai Delay, Overhead. 

Load Balancing 
Backup Reluibilily 

CoocurreoJ Boudwidib. Rdiabiliiy 
CoDcurreiu Reluibiliiy 

CoDcurreoJ ReluibiUiy, 

Reiiounx (JiilizauoD 
Coocurreiu Energy 

Concurrcni Reliabrliry 
Concurreni Robu&ineas 


Backup ReliabiliTy 

Backup Delay, ReliabilUy 

Concurmii Deby, ReliabilUy, 

Load Balancing 

Concurreni Energy, Load Balancing 
Cuncurreni Delay, Reliobiliiy. Ji tier 


SDMRfTT?! 

2008 

Source Routing 

Cuncurreni 

Load balancing. Bond- 
width. Inierferenc'e. Pad) 
Go*ene«s 

Muliiputh Routing Reliability 

rrra 

2008 

Overlay Routing 

Backup 

Reliability 

Mobile tnAid 

2010 

2014 

Hop-by-Hup 

SDN-bosed 

Cuncurreni 

Cuncurreni 

Delay 

Reliability 

Wtrtiesn Mesh .Vejvunbr (WU^s) 
Weighted Interference Multi¬ 
path tWlM)(^ 

Irnnt MiilltrViriling Mullioalb 

2007 

Source Routing 

Concurreni 

Interference, Delay, 
Reliability 

2007 

Source Routing 

Concurreni 

Throughput 

(JMM) 

wmSDN"^ 

2013 

SDN-based 

Backup 

Load Balancing, 
Resource Uiibiaiion 

Camidve Radio i\el*eorks fCRMs) 





Beltagy ei al. 

2011 

Hop-by-Hup 

Backup 

Reliability. Path Close- 

Alieial [Tzsl 2015 

Vehieulohanioc NetwoHu (VANETsi 

Hop-by-Hop 

Backup 

Reliability 

Huong etal.[^ 

2009 

Hop-by-Hup 

Cuncurreni 

Delay. Imetferenc'e. 
Reliability 

LIAITHON 

Software dedned VANETb 

2012 

Hop-by-Hup 

Cuncurreni 

Load Balancing. Inier- 
fereoce. Path CIuseoeM 

2014 

SDN-bosed 

Backup 

Reliability 

Penoaal Area Nerworki fPANtI 





IEEE 802 I54jl20| 

2011 

Hup-by-Hop 

Coocurrem 

Delay. Reliability 


Modifier D5R 10 support mullipalhing. 

Uses a proPing-bosed mechaniam lo provide rauliipaih support in D5R for load balancing. 
Consifucu iwu maximally disjoint paiJu, wiih one being the smallest delay path, in an 
00 -demand fashion. 

Modifies AODV (o consuiici multiple link-disjoini pailis using the adverliied hop couai 
mfonnaiion. 

Provides a OoS-aware mulu-paih dynamic source rouiiog protocol for MANETs. 

Modifies AODV to support nuilliple node-disjomi paths by introducing a few cvrUnHeil 
reiiuble nodes in llie lopology 

Circumvent frequent link breakdowns by caching la few) data packets along with aJienuue 
route information ai nodes distributlvely. 

Provides two algonthms to construct 2-Unk-disjoini and k-itode-disjomi pailis to conserve 
energy in dense and sparse networks, respectively. 

Sends data in the form of shufTt over multiple indepeivdent paib« lo ensure confideniialiiy. 
Provides reol-iime efTicieni video streaming using multiple paths for unicast and multicast 
traffics. 

Provides adaptive muliipaibing in AODV based on signal suvngth received ai nodes 
Consiructs node-disjoint muliiple paths for a source-desunation paic 
PresenUi an analytical model for multipaih rouiiog in MANETs for optimal load balancing, 
os well as lo address delay and reliability issue.«. 

Modifies OLSR to support muhipatbing 

Provides a hybrid muliipulhing algoriihni with reacuve path setup, as well as proactive path 
prul^g and majmenance. bused on on tini-cofcmv opiimlzuilvn frames ork. 

Enublen u source node lo ciilculiiie multiple nputiolly disjoint puihn based ihe infurmaiion 
esiracicd from route reply pockets from the desiinution node. 

Aiulyaes the reliability of multipath routing prutocoh bused on un overlay graph concept 
fromewuik. 

Jointly addresses ihe issues of multipath routing and deferential delay 

Use« a central controller in a SDN fashion to perform multipath routing in MANET. 


Selects multipath ba.sed on ^xei/ihiedmuiual-iHierferrrue metric in s roulti-cbanncl 

multi-radio environment. 

Spills traffic among differeni channels ai different times over multiple paths. 

Selects Internet gateways of a mesh network in a round robin fashion to balance outgoing 
traffics using a central controller. 

Consiiucu multiple routes to minimise the effects of PU acuvity based on a clvseness 
metric. 

Proposed a learning automata based muitlpathing solution for multicast routing in CRNs. 

Consiiucu node-disjoini multiple puihs bicsed un the mutual mterferene'e of pathii. 

Provides a receiver-initiated video »ireaming multipath »olution in which neJes comprising 
the multiple paths ore chunen so that they have minimum coupling^closeness. 

Deploys VANETs in u sufiware-defined environment for reb ability enhancement and 
power moiugemeni using a central controller, 

Chooses a pareni node oppoitumsticolly to enable multipathiog by combming iptf 
802.15.4's MAC layer c/urier-rrre method with RPL\desfuujiton~orierUcd directed acyclic 
graph (DODACt 













are many challenges involved in integrating source routing, 
multi pa thing, and global Internet routing [ fT?) . A major imped¬ 
iment being the rigid inflexible policies of various ASes for 
interdomain routing [ [T?) — e.g., it is typical for BGP to ignore 
potential paths suggested by a source routing scheme [ [T3| — 
which makes the control of the end-to-end routing behavior 
difficult for end hosts or even any single AS. There are ongoing 
efforts to devise new techniques to adapt BGP to implement 
source routing while supporting the provisioning of multiple 
paths through ASes (7?) , (7?| , With various trends on the 
Internet (such as multihoming} nudging the Internet towards 
multipathing. there is also an increased interest in adapting 
interdomain routing protocols for multipath. Different works 
have attempted to develop new techniques based on the legacy 
interdomain routing protocols that we describe next. 

Feedback-based routing (FBR) is an alternative ap¬ 
proach to interdomain routing. In contrast to BGP, this ap¬ 
proach differentiates routing information into its structural 
components (that deals with the existence of links) and 
dynamic components (which refers to the quality of these 
links). In this approach, the core routers of the Internet 
are only responsible for disseminating structural information, 
while the routing decisions are taken at the edge based on 
dynamic end-to-end performance measurements in addition to 
the structural information. FBR is multipath-based since each 
access router computes two paths to every advertised network 
prefix (that differs from each other as much as possible) based 
on the structural information it receives. The access router 
then monitors the route quality, and uses the belter path to 
carry its traffic. In case the access router finds that both the 
routes have an infinite RTT, then two new routes are computed. 
Since the core/ transit routers do not compute routes (and only 
disseminate structural information), FBR scales better than 
BGP. 

Tlie caveat of source routing is the overhead involved in 
carrying the routing information in the packet itself. Various 
approaches have been proposed for this. LI PS IN proposes 
putting a Bloom filter into data packets in source routing 
style as a forwarding fabric for multicast traffic for publish/ 
subscribe Internets |122| . MPLS can be used for explicit 
routing, which is similar to routing, by using labels avoiding 
the overhead of carrying the information of the entire path in 
the packets. The BANANAS architectural framework P?) , on 
the other hand, works by encoding paths using a short hash 
(called the PathID) which is a sequence of globally-known 
identifiers. 

In the BANANAS framework j4^ — which can be deployed 
in both intradomain or interdomain settings—is an attempt 
to make conventional routing protocols (such as OSPF, ISIS 
and BGP) capable of exploiting the path diversity of the 
Internet. The need of specifically constructing multiple paths 
is eliminated by generating a short hash called a Pal hiD 
using a globally accessible sequence of identifiers. A path is 
encoded as PathID using a hash on a sequence of identifiers 
(or any non-null pennutation of such sequence) that can be 
the IDs of routers, link interfaces, link weights, AS numbers 
(used in the case of interdomain routing), etc. The global 
significance of these identifiers distinguishes BANANAS from 


the conventional MPLS and ATM architectures. Routers that 
are multipath-enabled can calculate a PathID with the global 
scope of the above-mentioned identifiers in view. This feature 
makes the routers independent and autonomous with respect to 
the setup of multiple paths without the need of explicit control 
signaling protocol for a specific path setup. 

Multipath Inlerdomain Routing (MIRO) is a BGP- 
based interdomain multipath routing framework that is back¬ 
ward compatible with BGP and incrementally deployable. 
In MIRO, the default routes are learnt through the BGP 
protocol, and arbitrary pairs of domains can perfonn AS-level 
negotiations to add additional paths tailored to their specific 
needs. MIRO provides more flexible control of traffic to the 
intennediate ASes with reasonable overhead. This is done to 
promote path diversity and improve path resilience in inter- 
domain routing. The main features of MIRO include (i) AS- 
level path selection; (ii) negotiation support for alternate routes 
between two arbitrary ASes (that may not be adjacent); and 
(iii) support for policy-driven export of alternate routes by the 
responding AS during negotiation (through which transit AS 
have greater control over which traftic enters their networks). 
Although MIRO is similar to overlay routing in that it uses 
runnels, it differs from overlay schemes because it allows 
paths to be selected in the underlay through the cooperation of 
intennediate ASes. The dift'erence between MIRO and mo.st of 
the other source routing works is that, apart from providing end 
hosts or the edge routers with routing control, MIRO allows 
control over path selection by intermediate ASes. 

Tlie **new Internet routing architecture” (NIRA) is a 
source-routed interdomain multipath routing protocol in which 
the end host has more flexible control over the sequence 
of ASes to be traversed by its packets. By providing the 
end-host control over the adopted end-to-end path, NIRA 
can help foster healthy ISP competition leading to enhanced 
services and improved end-to-end performance and reliability. 
NIRA provides users/ end hosts with more control over the 
chosen routes without running a global link-state (LS) routing 
protocol, and thus NIRA can provide a user with a route with 
lower control overhead. With NIRA, a user only specifies a 
source address and a destination address to choose a specific 
interdomain path comprising multiple ASes, and the path 
is altered when the source or the destination addresses is 
changed. This technique is efficient because of its faster fail¬ 
over action and lower control overhead. 

The concept of routing deflections (57) is applied at the level 
of domains in pathlet routing (3T| . In pathlet routing, domains 
adverti.se multiple paths or path lets to a source located in 
another domain. These pathlels afford tlie source with the 
ability to exploit the advertised path diversity through route 
deflections. More abstractly, pathlet routing allows source 
routing over a virtual topology. Using the concept of routing 
over virtualized topologies, network operators can declare their 
routing services expressively in a flexible fashion. In particular, 
using pathlet routing, the routing styles of BGP, source routing, 
MIRO, LISP and NIRA can be emulated [|5T1 . 

AMIR is a multipath interdomain routing protocol proposed 
in (5^ that establishes multiple paths and deploys source 
routing at AS level. In this scheme, the multiple paths are 
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established around a primary path that is usually constructed 
by a policy-compliant traditional interdoraain routing proto¬ 
col. This is achieved by negotiating path provisioning with 
neighboring ASes. Tlie participating ASes are called imilti- 
patli service agents (MPSA) and these agents impose charges 
for providing topological information for the computation of 
multiple paths through them. Data is routed over these ASes 
by a multipath service fonvarding agent (MSFAi embedded 
at the edge routers of such ASes. Paths are set up and the 
data is forwarded in a source routing fashion using MSFAs 
of MPSAs. AMIR enhances application-level user experience 
with lower overhead incurred in path setup at the AS level. 

2) Source Routed Multipatlung (Wireless Solutions): There 
are various source-routed multipath routing proposals for 
wireless networks—with the major bulk of work being done 
for mobile ad-hoc networks (MANETs) and wireless mesh 
network (WMNs). In this subsection, we will cover the wire¬ 
less networking conligurations of wireless sensor networks 
(WSNs), MANETs, and WMNs. 

Source-Routed Multipath Protocols for WSNs: 

WSN multipath routing protocols have rarely adopted 
source-routed solutions to avoid the scalability problem that 
may arise by overburdening the potentially energy constrained 
source node. The major bulk of multipath protocols proposed 
for WSNs aie based on the hop-by-hop routing paradigm 
instead that will be covered in Section [Tl-BI As an example of a 
source-routed WSN multipath solution, we will discuss the H- 
SPREAD (95) routing protocol that facilitates the construction 
of multiple node-disjoint paths towards the base station in a 
distributed manner. In such a setting, only a few paths can be 
compromised if some malicious nodes appear in the network. 
If the next-hop node of a packet at an intermediate node is 
unavailable, then this node redirects the packet (in a source 
routing manner} to an alternate next hop which is part of an 
available backup path, thus enhancing the reliability of a data 
transfer. It is only when there is no available alternate path 
that a packet is dropped at a node. 

Source-Routed Multipath Protocols for MANETs: 

Most of the routing protocols proposed for MANETs are 
reactive/ on-demand. On-demand routing protocols compute 
routes when they are needed for communication and are 
better suited to dynamic topologies such as MANETs. Such 
an approach also avoids the extra overhead associated with 
proactive routing (due to the periodic exchange of control 
packets). In on-demand routing, routes are discovered through 
the flooding of specific control packets by the source that are 
used to gather the topological information. This infonnation is 
then collected using a reply packet from the destination. The 
legacy protocol for MANETs called dynamic source routing 
(DSR) |123[ works on the same principle. After computing the 
route, a source node embeds the route information in a packet's 
header. These packets are then forwarded by the intermediate 
mobile nodes to the eventual destination. 

In (59) . Nasipuri et al. have modified DSR so that it can 
support the provisioning of disjoint multiple paths. Two flavors 
of multipath routing are presented: in the first approach, only 


the source node is provisioned with multiple paths; while in 
the other, both source and intermediate nodes (that make up 
the primary path in the normal operation of DSR) are also 
provided with multiple disjoint paths around the primary path. 
Tliis work makes an effort to reduce the control overhead 
incurred by the conventional DSR in the route discovery 
process. This control overhead is a consequence of flooding 
the query packets in the network. The reduction in the control 
overhead is due to the provisioning of multiple backup paths. 
So. in the case when the primary path fails, the traffic is 
switched to one of the backup paths without rediscovering 
the route all over again. 

Another similar work based on modifications in DSR is 
called multipath source routing (MSR) that is proposed in [ |4T) . 
Multiple paths are computed using the conventional DSR’s 
route discovery and maintenance operations. In this work, 
a probing-based mechanism is used to achieve a balanced 
load over multiple constructed paths. This mechani.sm probes 
multiple paths for their RTT responses and the traffic is load 
balanced accordingly to control congestion and reduce end-to- 
end delay. 

Two maximally di.sjoint paths are created to balance the 
input traffic over them. One of the paths constructed is the 
conventional shortest path. This process is done In an on- 
demand fashion. This work is called split multipath routing 
(SMR) 111Q[ and it performs belter than DSR in terms of 
end-to-end delay. Traffic is distributed over the multiple paths 
on a per-packet manner. This technique, although effective for 
load balancing and controlling congestion, can create a packet- 
reordering problem at the destination that makes its use with 
TCP problematic. 

Multiple independent paths are deployed for the purpose 
of improving secrecy and security in (45) . This method is 
called SPREAD. In this method, a secret message is divided 
into multiple parts called shares. Then, using secret (threshold) 
sharing methods, these shares are routed over multiple inde¬ 
pendent paths towards the destination. An adversary node has 
to compromise at least a certain number of shares (a specific 
threshold number) out of the total to completely decipher 
the original message. To enhance the security further, the 
shares are forwarded over multiple independent paths towards 
a destination. In this case, an attack on all the paths must be 
made by salvaging at least the threshold number of shares to 
decode the original message. By this way, even if some of 
the nodes along the paths are compromised, the message as a 
whole is protected. 

Physical distance between the discovered paths in a mul¬ 
tipath routing protocol is critical in the case of wireless 
networks. This is because of the mutual interference between 
the paths that can degrade the communication quality. Spatially 
disjoint multipath routing protocol (SDMR) is proposed in 
P 15| . A source node floods route request packets towards its 
destination. On the way back, in the Route Reply packets, the 
intennediate nodes, between the source and the destination, 
embed their link-state information (containing information 
about one-hop neighbors) in these packets. This helps the 
source node to obtain a partial graph of the whole network 
and choose paths that have the largest spatial disjointness. 
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The packets are then forwarded using source routing in a 
round robin manner over the discovered multiple spatially 
disjoint paths. The goal of this scheme is to reduce mutual 
interference between the discovered paths to gain the benefits 
of load balancing and bandwidth aggregation in MANETs. 

Source-Routed Multipath Protocols for WMNs: 

In |118| , reliability and interference minimization are 
studied in a multi radio multichannel WMN. In this work, 
non-disjoinl multiple paths are considered, and reliability is 
achieved through packet duplication over tliese paths. A met¬ 
ric, called weighted interference minimization path selection 
metric (W/M), is introduced for interference minimization 
among the paths. This metric and the accompanying algorithm 
consider interference among paths and neighboring nodes 
using the same radio channel. It has been shown that this tech¬ 
nique, where non-disjoint multiple paths are used, enhances 
reliability, interference minimization and end-to-end delay as 
compared to the techniques where only maximally disjoint 
paths are considered. 

In (45) , joint multi-channel multi-path control (JMM), 
which is a cross-layer design involving the multi-channel link 
layer and the routing network layer, is proposed to enable 
multipath routing in WMNs. In this study, the time horizon is 
divided into slots, and then different channels are attributed 
to different time slots. By this way, the contending input 
traffic is balanced at different times over different channels 
and different paths. As a result, the end-to-end throughput 
performance is enhanced. 


B. Hop-b\-Hop Routing 

In a hop-by-hop routing scenario, routing decisions are 
taken, as opposed to source routing, in a distributed or per- 
hop fashion. Each node along a path processes incoming 
packets and forwards them by consulting a routing table stored 
in its memory. Hop-by-hop routing is especially popular for 
multihop and infraslruclure-less environment such as WSNs 
due to its decentralized and robust nature. Hop-by-hop is by far 
the most famous and widely adopted technique in IP networks. 
Link state (LS) and distance vector (DV) routing schemes are 
the most popular techniques that have been adopted in such 
routing scenarios. In the rest of this section, we review works 
that deploy this routing technique in different network types. 

/) Hop-by-Hop Midtipathing (General/ Wired Solutions): 
A link state-based algorithm is proposed by Vutukury and 
Aceves in (55) to calculate loop-free multiple paths with 
minimum dependance on the underlying physical network 
topology. In (57) and [ [55) , Vutukury and Aceves present a 
distance vector-based algorithm to address the same problem 
of finding loop-free multiple paths. In all of these works, the 
multiple paths are provisioned based on a special condition 
called loop-free invariant (LFI) (which is described in ]56|). 
When a routing algorithm satisfies this condition, then the 
loop-free ness at every instance is ensured. 

In (73) , a family of protocols called maximum alternative 
routing algorithm (MARA), which is based on a directed 
acyclic graph (DAG), is presented. Using the DAG approach 


ensures that loop-free paths are constructed. Based on a 
network topology, the DAG approach constructs a graph with 
the maximum number of multiple paths within a minimum 
amount of time subject to a throughput constraint in order to 
improve the reliability of communication. 

Hashing enables a node to select a path from a set of 
available paths while forwarding packets along a path. Two 
hashing schemes, namely direct hashing and table-based hash¬ 
ing, are studied in (59) for the purpose of packet forwarding 
in a multipath routing scenario. In direct hashing, a hash 
function is performed on a packet’s header, and the output 
is directly mapped to one of the paths from the set of 
constructed multiple paths. In table-based hashing, on the other 
hand, the output of a hash function is first put in one of 
the bins (with the total number of bins potentially differing 
from the total number of available paths) and the bins are 
then mapped to multiple paths. In the work of Cao et al. 
| |59) , five 16-bit cyclic redundancy check (CRC}-ba.sed direct 
hashing techniques and an XOR of source-destination IP- 
ba.sed and table-based hashing technique are developed for 
balancing the load over equal-cost multiple paths (ECMP). 
ECMP is a simple method for provisioning multiple paths 
that has been incorporated into many conventional routing 
environments (such as OSPF, MPLS and ISIS) (55), ||5D). 
ECMP is explained in detail in Section Ill-Cla An interesting 
feature of table-based ha.shing is that it can be used to balance 
load over unequal path weights (as described in (55) ) at the 
cost of requiring more slates than direct hashing. 

Dispersity routing enables concurrent transmission of data 
over multiple paths (53) . There are two different flavors 
of dispersity routing. A node can either send distinct data 
packets over multiple available paths for load balancing, or 
send duplicated data packets over multiple paths for improved 
reliability and resiliency. 

R-BGP is an extended version of the traditional BGP routing 
in which multipath routing is applied to hierarchical networks 
comprising different ASes (77) . In addition to the primary 
paths, the additional multiple paths serve as backup paths, 
and all of them aim to maintain connectivity even in the face 
of multiple failures subject to the compliance with AS policy. 
In (53) , an advertisement mechanism is proposed for BGP to 
advertise information regarding multiple paths at the AS level 
| |53) while minimizing control overhead with the objective of 
providing path diversity as a means to circumvent failures. In 
| |75) . BGP is enhanced to calculate loop-free multiple paths. 

Yet another multipath routing (YAMR) is a multipath rout¬ 
ing protocol that constructs multiple paths and establishes con¬ 
current transmission in hierarchical networks (23 It renders 
reliable communication against any single interdomain link 
failure. Compared to BGP, it incurs lower control overhead 
by isolating failures and keeping the whole Internet oblivious 
of their occurrences. This technique of containing the failure is 
termed as hiding. This technique combined with multipathing 
helps YAMR to converge with a lower control overhead than 
BGP while demonstrating enhanced reliability. 


2) Hop-by-Hop Multipathing (Wireless Solutions): In this 
section, we will present a summary of hop-by-hop multi- 
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pathing protocols that have been proposed for wireless net¬ 
works. 

Hop-by-Hop Mitllipalh Protocols for WSNs: 

Conventional single-path routing schemes put all traffic 
on a precomputed set of links, thus quickly draining the 
energy of the intervening network elements. In multipath 
routing techniques ]1Q5| , on the other hand, network traffic 
is distributed over multiple paths with load balancing and 
concurrent transmission, thereby avoiding overburdening a 
single set of network elements and enhancing the network 
lifetime (a critically important feature in WSNs). 

The provisioning of multiple paths in a wireless scenario is 
prone to inlra- and inter-path interference. In (55) , a radio 
disjoint multipathing method is proposed. This technique 
constructs multiple paths in an incremental fashion keeping in 
view the congestion and interference effects, so it helps to im¬ 
prove bandwidth gains among bandwidth-hungry multimedia 
applications. In the first instance, only one path is established; 
and upon receiving feedback on network congestion, other 
paths are established in an incremental fashion. However, 
when high interference is observed, sensor nodes may be put 
into passive mode to reduce the number of multiple paths, 
and with that the amount of interference as well. Overall, this 
technique improves throughput and energy efficiency but at 
the cost of higher control overhead. 

Another technique called meshed multipath rouiing (M- 
MPR) with selective forwarding is proposed in (5?) , (54) 
to incorporate forward error correction (FEC) and selective 
forwarding into multipath routing for better reliability in order 
to improve throughput performance. In selective forwarding, 
a packet is forwarded over the path with the best down¬ 
stream wireless channel quality, with acknowledgement driven 
retransmissions eliminated by using FEC mechanism, which 
facilitates efficient load balancing with lesser control overhead. 
It has been shown that M-MPR provides higher tliroughput 
than its disjoint multipath counterparts. 

Yang et al. (75) have proposed a nerwork coding-based 
reliable disjoint and braided multipath routing protocol. In the 
disjoint rouiing approach, multiple disjoint paths are created in 
a hop-by-hop fashion and routing information is stored locally 
on a node which does not take end-to-end path in view. A 
node groups its neighbors based on their respective distances 
from a destination node in terms of the number of hops hence 
aiding the load balancing process, in which each packet is 
forwarded after an election at a node with the path with lesser 
number of hops towards a destination being selected. Each 
node calculates a few logical paths as backups, which may not 
be completely disjoint, and hence are called braided multiple 
paths. Reliability is introduced in this work by the deployment 
of network coding at the source and the intermediate nodes. 
Tlie encoded packets using this technique can only be decoded 
at the respective destination nodes. Simulation results show 
that multipathing with braided paths works better in terms of 
reliability and load balancing in WSNs. Braided multiple paths 
are considered in another study (95) for improving reliability 
and energy efficiency. It is observed that the braided path 
technique can avoid 50% more isolated failures while also 


having a lesser control overhead when compared to disjoint 
path techniques. 

A geographical approach known as geographical power 
efficient routing (GPER) for multipathing has been proposed 
for WSNs (55) . This protocol enables individual nodes to 
select their own transmission ranges depending upon their 
respective residual energy resources thus enhancing network 
lifetime. This act also helps in preserving the energy resources 
of a passive listening neighbor. As a further improvement, 
multiple paths are provisioned on a per-node basis. This 
approach is adopted so that the overall communication can be 
distributed over a wider area of a network and hence saving 
a particular set of network elements from being overwhelmed 
and deprived of their energy resources sooner than the rest of 
the network. It has been observed that by doing so enhances 
the network lifetime by up to 30%. 

Although the traditional focus of WSN research has been 
on energy efficiency, the requirement of security is also very 
important—especially for environments such as military or 
healthcare. Stavrou et al. |124| have surveyed security-related 
issues. A threat model that covers the behaviors of malicious 
users is presented, and the secure multipath routing protocols 
for WSNs are categorized according to their specific opera¬ 
tional capabilities in achieving security-related objectives. 

The problem of Joint fault and intrusion tolerance is con¬ 
sidered in |IQ8| . A distributed and in-network fault-intrusion 
tolerance protocol called secure and efficient intrusion-fault 
tolerant protocol (SEIF) is developed. This technique renders 
nodes to take decisions in regards to resilience and reliability 
that come with independence from base station. By this way, 
the time incurred to detect an intrusion is reduced. This 
security mechanism is incorporated into a multipath routing 
protocol called sub-branch multipath routing protocol (SMBR) 
to improve network discovery time and energy efficiency, 
as well as to reduce control overhead as only one message 
per node is required to construct a network topology. H- 
SPREAD (^ is a routing protocol for WSNs that facilitates 
the construction of multiple node-disjoint paths towards the 
base station in a distributed manner. In such a setting, only a 
few paths can be compromised if some malicious nodes appear 
in the network. 

Hop-by-Hop Multipath Protocols for MANETs: 

Ad-hoc on-demand multipath distance vector routing 
(AOMDV) [ |55) is one of the famous modifications of AODV. 
AOMDV provisions multiple link-disjoint paths between a 
source-destination pair in an on-demand manner. Another 
AODV modification is presented by Ye et al. in (57) . Reliable 
nodes, in terms of energy resources, are deployed in the 
network topology. These nodes are used to construct reliable 
multiple paths, along with other unreliable nodes, to ensure 
routes are more reliable and less prone to failures and out¬ 
ages in MANETs. An algorithm is presented to control the 
placement and trajectories of the reliable nodes so that these 
nodes are highly likely to be part of a path that a normal 
(or unreliable) node constructs. An important discovery made 
in this work is that there can only be a few disjoint paths 
between a source-destination pair in a MANET—with the 
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number of disjoint paths decreasing as the distance between 
communicating pairs increases. This discovery motivates the 
use of reliable nodes in such networks. 

MP-AODV is an adaptive multipath routing scheme based 
on the conventional AODV routing scheme proposed in [ [55) . 
Primary paths are selected based on the received signal 
strength. Alternative paths are then constructed by the ex¬ 
change of, so called, source generated liearibeaT messages 
among nodes in a MANET. These messages help a source 
to gatlier the mobility information of nodes that make up 
the multiple (backup) paths towards a destination. One of 
the main objectives of this work is to enhance the reliability 
of communication in MANETs. Another work (59) presents 
node-disjoint muhipath routing (NDMR) protocol that also 
makes use of backup paths, and improves upon the legacy 
protocols, namely AODV and DSR, In this work, the issue 
of reliability in terms of packet delivery ratio is addressed. 
Besides reliability, this protocol is also efficient in terms of 
end-to-end delay and control overhead. 

A technique called packet-caching is proposed by Valera 
et al. in | |5T) to ameliorate congestion and packet losses with 
the provision of multiple paths. In this work, caching and 
multipath (CHAMP), which consists of cooperative packet 
caching and shortest multipath routing, is proposed to cater 
for route breakdown, which is a common yet important issue 
in MANETs. In CHAMP, each node has a buffer in which it 
stores/ caches data packets that it encounters during a normal 
operation. Infonnation of alternate routes is also cached at the 
same node to be used when the downstream node (relative 
to this node) fails to forward a data packet or somehow 
malfunctions. If such a situation occurs, an upstream node 
forwards the dropped packet from its cache along an alterna¬ 
tive path. This technique performs belter than DSR and AODV 
in temis of end-to-end delay, packet delivery ratio and control 
overhead. In another work (51) . reliable adaptive multipath 
provisioning (RAMP) is proposed for reliable adaptive mul¬ 
tipath provisioning to meet bandwidth and differential delay 
constraints. The problem of differential delay is particularly 
important for the multipath routing .scenario. When traffic is 
dispersed over multiple paths, then different paths may offer 
different amount of delays, and as a result the traffic reaches 
at a destination at different times. This difference in delay 
among multiple paths is called the differential delay. Traffic is 
allowed to traverse over multiple paths, which may be di.sjoint, 
so that communication outage due to a single link failure can 
be avoided. 

An energy saving protocol for MANETs is proposed by 
Srinivas and Modiano in (5P) . In order to conserve energy, 2- 
link-disjoint and k-node-disjoint algorithms are proposed for 
dense and sparse networks, respectively. A set of algorithms 
are developed to find node- or link-disjoint paths in a given 
network graph so that transmission energies of nodes are 
minimized. An interesting observation has been reported in 
this work, that is the incremental energy of each newly added 
link-disjoint path is decreasing due to the wireless nature of 
the underlying network. 

Tliere has also been work done on multipathing for a 
specific configuration of MANET known as vehicular ad- 


hoc networks (VANETs). In (75) , a node-di.sjoint concurrent 
multi pa thing protocol is proposed for VANETs to improve the 
overall packet delivery ratio even though the relaying nodes 
may have unknown speeds. In another video streaming work 
for VANETs |119| , location-aware multipath video streaming 
(LIAITHON) is proposed for urban vehicular networks. In 
this protocol, a receiver initiates a video streaming process 
by sending a request to a source. The source then transmits 
the video stream over the intermediate nodes that are well 
away from each other to reduce inter-patli interference. The 
selection of such nodes is carried out using the geographical 
location information of these nodes (with respect to the source 
node). 

Tliere have also been biologically inspired routing protocols 
proposed in MANETs, particularly AntHocNet, which is an 
ant colony optimization (ACO)-based multipath routing proto¬ 
col j7^, j7^. The ACO-ba.sed AntHocNet protocol has both 
a proactive path probing component as well as a reactive path 
setup/maintenance component. It is shown in |75) , (TT) that 
AntHocNet outperfonns AODV in terms of delay, jitter and 
packet delivery ratio, while having the drawback of being more 
expensive in terms of its control overhead. 

Hop-hy-Hop Multipath Protocols for CRNs: 

In (75) , a multipath routing scheme for cognitive radio 
networks (CRNs) is deployed by introducing a metric that tells 
about the route closeness of multiple paths, which depends on 
the physical distance among the multiple available paths. In 
CRNs, an unlicensed or secondary user (SU) can access the 
licensed radio .spectrum if it is not being used by the licensed 
or primary users (PUs). Since the PU has higher priority over 
SUs, the arrival of PU precludes SUs from accessing the spec¬ 
trum and makes the routing problem challenging. The paths 
with less clo.seness are preferred because such paths are less 
prone to the PU activities due to the greater distance among 
these paths. In another interesting CRN-based work fll25| , 
contrary to most of the multipath routing works presented in 
this paper (that have addressed unicast routing), Ali et al. have 
proposed a multipath-based approach to multicast routing in 
CRNs. The proposed approach makes use of learning automata 
to select active and backup paths to develop multicast routing 
protocols that is minimally dismpted by PU activity. 

Hop-by-Hop Multipath Protocols for PANs: 

There have also been works done on multipathing for 
wireless personal area networks (WPANs). Multipathing is 
realized over the MAC layer of the low-rate WPAN (LR- 
WPAN) standard IEEE 802.15.4 in fll20| . This realization is 
achieved by selecting multiple parent nodes in an opportunistic 
manner at a lime. This is done by making IEEE 802.15.4’s 
MAC layer’s cluster tree to operate in harmony with the desti¬ 
nation oriented directed acyclic graph (DODAG) as proposed 
in routing protocol for low-power and lossy networks (RPL) 



C. Overlay Routing 

We have earlier seen that source routing equips end hosts 
with the ability to deal with any failures arising along a route 
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itself. The source can thus either identify an end-to-end route 
itself as is done in the “source routing*’ paradigm, or it may use 
the “overlay routing** paradigm to choose some intermediate 
hosts to route through in order to avoid performance and relia¬ 
bility problems that may exist on the direct route. An overlay 
network is essentially a virtualized logical network built on 
top of a physical network with tunnels interconnecting edge 
devices. The overlay network appears to the nodes connecting 
to it as a native network with the possible existence of multiple 
overlays on the same underlying physical infrastructure (which 
allows support for multi-tenancy). Overlay routing can be used 
to override suboptimal routing in the substrate networks, and 
can be used to support more flexible self-driven routing control 
of multihomed end hosts. In this section, we will discuss the 
overlay routing paradigm. 

}) Overlay Routed Midtipathing (General/ Wired Solu¬ 
tions): Source routing is not supported in most of the In¬ 
ternet routers. The most common technique used to support 
multipath on the Internet is to maintain an overlay network 
for connecting nodes and supporting source routing. Overlay 
routing is especially popular on the Internet for supporting TE 
and mullipathing. 

Multihoming for overlay networking is deployed in [ [9?) , 
and a new multihomed overlay netM’ork (MONET) protocol is 
proposed. The purpose of this work is to increase the avail¬ 
ability of websites to users by creating multiple paths to web 
proxies. A method called way point selection is used to choose 
a subset of available paths leading to the proxies in order 
to access websites. This method decides the order in which 
each path is chosen to forward packets towards a particular 
destination server depending upon the success statistics of 
the paths. In another work, Andersen et al. [ [7?| proposed to 
enable an end user to switch between actual network paths 
and those constructed by an overlay network. Tliis incentive is 
also available to the applications depending on their network 
requirements. In order to enhance bandwidth availability to 
meet application requirements, an overlay networking-based 
multipath solution is presented in (51) . 

Multimedia streaming is another important application in 
which reliability is important for seamless transmission. Mul- 
tipathing reduces Jitter and packet loss for efficient and reliable 
multimedia streaming. A coding technique called multiple 
description coding (MDC) p27| , p28| is deployed to enable 
multipathing for streaming media. In this coding technique, 
streaming media from content distribution networks is divided 
into multiple independently decodable streaming flows p29| . 
Packets from these multiple streaming flows are then trans¬ 
mitted over multiple disjoint paths. Many works base their 
multipath techniques on the streaming media using MDC. 

In another overlay routing scheme. Savage et al. proposed 
the Detour overlay network 0 — comprising geographically 
distributed routes interconnected using tunnels (which act as 
virtual point-to-point links) over the commodity Internet— 
to be used for better end-to-end performance by routing 
around performance bottlenecks. Any host using the Detour 
network directs its outbound traffic to the nearest Detour 
router with the packets taking an exit at a point close to the 


destination. Detour uses a dynamic multipath routing approach 
with automatic load balancing and congestion avoidance for 
high performance. The router and congestion is controlled at 
the Detour routers—which are, technically speaking, still edge 
devices of the overlay network. 

In (50) , a quality of service (QoS) metric based on link 
correlation is presented for overlay networks. This metric, 
along with path correlation, is then used to select multiple 
paths for an MDC streaming video flow. It is observed that 
this overlay network-based technique improves SNR compared 
to the conventional link-disjoint methods by up to 3.2 dB. 
Another work in (^T) makes use of overlay routing and MDC 
for efficient and reliable video streaming. Based on MDC, 
different multipath streaming models have been derived. Then, 
based on these models, an optimum strategy is given to 
select an optimum set of multiple paths. This technique also 
improves the SNR by up to 6.07 dB as compared to the single¬ 
path and link-disjoint multipathing. 

Various researchers have studied how to support virtual 
overlay networks on top of the physical .substrate networks, 
with a few works exploiting multipath for efficiently support¬ 
ing virtual networks (VN) on the underlying substrate. Yu et al. 
[ |13Q| have studied the strategies that the substrate network can 
adopt to flexibly support virtual network embedding in which 
the virtual network requests are mapped to specific node.s/ 
links in the underlying substrate network. Yu et al. simplify the 
general problem of virtual link embedding (which is known 
to be computationally intractable) by allowing the substrate 
network to (1) split a virtual link over multiple substrate links, 
and (2) migrate paths to periodically optimize the utilization 
of substrate network. The related problem of multi-domain 
VN embedding with limited information disclosure is studied 
by Dietrich et al. |131| who proposed a VN embedding 
framework that enables VN request partitioning scheme while 
assuming only limited information disclosure. 

2) Overlay Routed MulTipathing (Wireless Solutions): In 
this section, we introduce various overlay multipath routing 
protocols proposed for wireless networks, specifically WSNs. 

Overlay Multipath Protocols for WSNs 

A multipath routing protocol called MMSPEED is 
proposed to achieve QoS guarantee. In this protocol, multipath 
routing is deployed in a distributed overlay manner to improve 
reliability while .satisfying the constraint of packet delivery 
speeds. Here, the packet delivery speed is equal to the ratio of 
one-hop average distance to one-hop average delay. Braided 
multiple paths are considered in (95) for the enhancement 
of reliability and energy efficiency. It has been observed 
that this technique works 50% better as compared to its 
disjoint counterparts in terms of isolated and patterned failure 
avoidance with low control overhead. The isolated failure here 
refers to those incidents in which nodes fail randomly and 
independently, while in the case of patterned failures, nodes 
belonging to a specific geographic region fail. 

D. SDN-based Routing 

In this section, we discuss SDN-based multipath routing 
proposals. SDNs allow networks to be programmable through 
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TABLE IV: Routing Metric Based Classification of Multipath Routing Frameworks 
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Traditional Routing Metrics Used For MuUlpaSh 

Delayi Latency. Delay refers to the average end-to-end latency experienced by a communicating source-destination pair. We are interested 
Id mimmizing latency whicii can be achieved by mimmizing oeiwod: congestion and alleviaung bottlenecks. Solutions includes concurrent 
multipathing for duipersmg the irafTic to avoid bottlenecks, and using backup paths for more rapid route rediscovery after failure. 

RcUahiUtfi Resiliency. Reliability refers to the ability of the networking protocols to work consistently as required. Failures can be 
circumvented by using backup paths, and higher packet delivery ratio can be ensured through concurrent muliipatbing over diverse links. 

Bandwidth/ Throughput. Throughput, or more informally bandwiddt refers to the amount of data ihai can be uansferred within a period 
of time. The use of uadiiional single-path routing schemes con result lo congestion boispois that act a.s a performance bottleneck. Load 
balancing and dispersion of network traffic using multiple paths ai tbe same lime offer higher bandwidth and throughput guns. 

Conttvl Overiiead: Control overhead refers to the number of control messages that need to be otchanged to compute and maintain multiple 
paths An efficient multipathing protocol minimizes control overhead. 

Energy Efficiency: Energy efficiency refers lu the extent die multipath technique conserves energy resource. Such a metric is important fur 
WSNs and MANBTs In which energy is a limited and scarce resource. Both concurreni iriUiHmnrfion and traffic engineering can increase 
network lifetime by dispersing the traffic over multiple paths. 
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the separation of the control plane and the data plane, as 
well as through the support of high-level control abstractions 
[ |153| , |154| . The centralized decision making approach at 
the SDN controller, along with the development of new 
sophisticated control abstractions, allows convenient control 
and management of networks, and can be used to implement 
multipath routing. 

1} SDN-Routed Multipathing (General/ Wired Solutions): 
Software-defined networking (SDN) is becoming increasingly 
popular for management of networks, especially in DC envi¬ 
ronments. SDN is often an enabling technique for the opera¬ 
tions of DCs. Rain Man is a SDN-based firmware architecture 
proposed in (55) . This firmware has a layer two (L2) routing 
mechanism that enables efficient multipath routing in DC 
networking. Another work called Hedera, with a global view 
of active flows, presents an adaptive and efficient multipathing 
approach based on SDN concepts (55) . In (54) , Micro-TE is 
proposed so that flows in a DC environment are scheduled 
using a central controller with the help of stochastic properties 
of network traffic and multipath routing. Full link utilization is 
achieved in B4 [ [55) using traffic engineering solution with the 
help of OpenFlow-enabled centralized controller and packet 
forwarding mechanism using multiple paths. OpenHow is a 
protocol for SDNs through which the control plane of a central 
controller taJlcs to (or programs) the data plane of the switches 
in the network. 

2) SDN-Rotiied Multipathing (Wireless Solutions): In this 
section, we introduce various overlay multipath routing pro¬ 
tocols proposed for the wireless networking configuration.s of 
MANETs, WMNs and VANETs. 

SDN-based Multipath Protocols for MANETs 

SDN concepts have been used in |117| lo realize a MANET 


infrastructure. It has been shown that carrying out routing op¬ 
eration through SDN enhances the reliability of such networks. 
Concurrent transmission of data packets has been studied as 
a use case to demonstrate that this technique increases packet 
delivery ratio at the expense of moderate control overhead 
increase. 

SDN-based Multipath Protocols for WMNs 
An interesting study called wireless mesh software-defined 
networks (wmSDN) is presented in m In this study, a mesh 
network comprises wireless mesh routers (WMRs) that are 
IEEE 802.11-based and OpenFlow-enabled. The data plane 
of these WMR.s is programmed via a centralized controller. 
Traffic balancing i.s carried out by the central controller u.sing 
an algorithm called gateway selection algorithm (GSA). In 
thi.s technique, the WMRs act as gateways for mesh network 
to forward traffic out of the network, and these are selected 
for Internet connectivity in a round robin fashion, where the 
least used gateway is selected first and so on, instead of u.sing 
the nearest gateway at all times to forward traffic out of this 
network. This is done in order to achieve an enhanced network 
utilization. 

SDN-based Multipath Protocols for VANETs 
A similar work is presented for the case of VANETs in 
[ (55) . In this work, routing decisions are controlled using 
a central controller in an SDN environment. Multipathing 
provides resiliency in the case of emergency scenarios, such 
as route failures, or for the high priority flows. To achieve this, 
the controller reserves a few paths dynamically as backups. 

£. Routing Metrics for Multipath Routing 

Different metrics have been used lo quantify the perfor¬ 
mance and quality of multipath routes, including some that 
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are direct extensions of the common routing metrics used 
in single-path routing (e.g.» throughput, delay, congestion) 
and others that are customized for the unique settings and 
requirements of multipath routing. We discuss both types 
next. A summary of the various metrics used for multipath 
routing is presented in Table [TV} The choice of routing metrics 
used by various wired and wireless multpathing works is also 
summarized in Tables UTI and HTTl 

I) Multipath Specific Metrics: The quality of multipath 
routing essentially depends on the independence and diversity 
of the paths chosen (so that an isolated fault/ failure is not able 
to affect all of the paths). We have divided multipath specific 
metrics into three types, namely those that measure: (i) path 
disjoinledness/ independence; (ii) interpath interference; and 
(iii) path closeness/ correlation. These metrics are related to 
each other—e.g., if we select multiple paths that are disjoint, 
we also select paths with lesser closeness/correlation. The 
similarity of these metrics stems from the fact that they all 
measure correspondence to the qualitative ideal of maximally 
independent/diverse collection of paths that do not share a 
common bottleneck. Each of these metrics has been used in 
various network configurations including wired and wireless 
networks (as can be seen in Table [TVJ. 

Path Disjoiwtedness/ Independence: As noted earlier, the key 
to benefiting from a multipath routing framework is to ensure 
that multiple paths support each other when needed. This is 
best ensured when all the paths are independent so that a 
congestion hotspot or a performance bottleneck is not able 
to simultaneously affect the different paths. Many proposed 
multipath routing works have utilized some variation of this 
metric for path computation to measure link- or node-disjoint 
paths that are maximally independent. Examples of works 
proposed for wired networks include |I32| , |133| ; while works 
proposed for wireless networks include totally disjoint paths 
[ |100| and maximally radio disjoint routing [ |551 for WSNs; 
SDMR |115| and SMR |110| for MANETs, and node-disjoint 
multipath and LIAITHON 31191 for VANETs. 

Interpaih Interference Minimization: Routing protocols can 
be designed to prefer paths that minimize interpath inter¬ 
ference. With reduced interpath interference, concurrent us¬ 
age of resources can be performed and the overall routing 
performance improves. A number of Interpath intereference 
minimizing multipath protocols have been proposed, both for 
wired networks (such as the MPLS-based dynamic constrained 
multipath routing approach (42| ) as well as for wireless net¬ 
works (such as the use of weighted interference minimization 
or WIM metric for WMNs plH| and the 1M2PR protocol for 
WSNs I fTOI I). 

Path Closeness/ Correlation: Path closeness/ correlation is 
very similar to path disjointedness/ independence metric—and 
indeed, it can be thought of as its variant. With multipath 
routing, we seek to minimize the metrics of ‘path closeness' or 
‘path correlation' to prefer multipath paths that are mutually 
more diverse. Such metrics have been used both for wired 
networks (SUI, |134| as well as for wireless networks (such as 
the SDMR ^lS j work for MANETs, and the work of Beltagy 


et al. for CRNs). 

2) Traditional Routing Metrics: Some applications might 
be delay sensitive, while the others put emphasis on reliability. 
Here, we present different classes of multipath routing proto¬ 
cols whose design considerations revolve around a specific 
metric. The metrics being discussed in this section include: 
(i) delay, (ii) throughput, (iii) reliability, (iv) control overhead, 
and (v) energy efficiency. 

a) Delay/ Latency: In many time-sensitive applications, 
the delay metric is of vital importance. In the case of WSNs, 
time sensitivity matters In the case of military, health and life¬ 
saving scenarios, such as a fire-detection network application. 
In such scenarios, using multiple paths concurrently can sig¬ 
nificantly reduce end-to-end delay. In concurrent multi pie-path 
tran.smissions, which serve as congestion avoidance mech¬ 
anism (59) , (90) . using efficient load balancing and flow- 
splilting techniques help to avoid bottlenecks contributing to 
lower end-to-end delay. Another aspect is in the usage of 
multiple paths as backups (59) , | [55) , (55) . In this case, when a 
primary path fails, the traffic is shifted to one of the alternative 
paths from a pre-calculated set of backup paths. In this way, 
the amount of time for route rediscovery is reduced. 

b) Band\vidti\/ Throughput: Bandwidth efficiency, or ef¬ 
ficient utilization of network resources, is among the most 
Important benefits of multipathing. Concurrent mulllpathing, 
with efficient techniques of load balancing and flow splitting, 
distribute the network traffic across a network. By this way. the 
network traffic does not overburden a single set of node.s and 
links as oppo.sed to the case of single-path routing ]155| . [ [54) . 
| |59) . The majority of existing .state-of-the-art multipathing 
protocols utilize bandwidth/ throughput as part of the goals 
In network performance enhancement. 

c} Reliability/ Resiliency: Packet delivery ratio is an 
important metric to measure reliability/ resiliency. A number 
of factors such as network congestion, interference, mobility 
make links/ nodes prone to failure. Ideally, network routing 
should be resilient such that in the case of the failure of 
the primary path, routing switches to an alternative backup 
path (if available) within a short period of time. In addition, 
using multiple paths in a concurrent way, the effects of 
bottlenecks and congestion can be reduced through packet 
duplication or redundancy. Many works discuss the issue 
of reliability, resiliency and packet delivery ratio in several 
networking applications (91) . | |95) , (72) . In the case of general/ 
wired networks, multipath routing is mostly used to provide 
reliability in both intradomain and interdomain scenarios | P9) . 
| |5T) . Multipathing provides reliable communications in WSNs 
fT95) , JT55) , CRNs (75), VANETs (75). IEEE's 802.15.4 
1(725) , and MANETs (35), (^, (^, (50, |fTT2) . 

d) Control Overhead: Implementation of multipath rout¬ 
ing is often more expensive than single-path routing due to the 
extra cost of control messaging imposed by discovering and 
maintaining more than a single path for a source-destination 
pair. A number of works have focused on minimizing control 
overhead associated with multipath routing techniques. An 
example is a multipath distance vector routing scheme called 
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Fig. 2: Summary of route computation techniques for wired networks 
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Fig. 3: Summary of route computation techniques for wireless networks 


MDVA shown in (58) . In addition to computing loop-free 
paths, this scheme uses the technique of diffusing computation, 
where a node reports its distance information to an upstream 
node after calculating it based on the distance information 
from its downstream node, and neighbor-to-neighbor synchro¬ 
nization which is more efficient in terms of control message 
exchange compared to distributed Bellman-Ford (DBF) and 
topology broadcast (TOPB) algorithms; specifically, it im¬ 
proves the convergence time after link failures. The control 
overhead routing metric is highly relevant to energy efficiency. 
To avoid drainage of energy resources, it is desirable to contain 
the number of control messages that need to be exchanged. 
Researchers have focused on this problem and have proposed 
routing schemes to limit the control overhead [ [?T| |, pQ3| ]. The 
routing metric of energy efficiency has been applied in WMNs 
ijSS), fnU\ and MANETs (^, fTTOI ^ 

e) Energy Efficiency: Energy is one of the main concerns 
in the design of routing protocols, particularly in power- 
constrained networking configurations such as WSNs and 
MANETs. Multipath routing, with the use of concurrent 
multiple paths, improves energy efficiency (95) , (5^ , [ [54) . 
In a legacy single-patli routing, the network resources of the 
primary path are drained much quicker because of the traffic 
load it bears. Using multipathing and concurrent transmission 
with efficient load balancing techniques, the network traffic 
spreads out ensuring that only a few particular network ele¬ 
ments are not overburdened. This provides not only bandwidth 
and throughput benefits but also energy conservation. 

F Summary' of Trends 

In this section, we identify the main trends observed on 
the basis of papers presented in this survey (which in our 
view provide a balanced representative view of the overall 
multipath routing literature). The trends of the commonly 
adopted routing approaches/ choices are noted below and 
summarized visually in Figure [^ for wired networks, and in 
Figure]^ for wireless networks. 

• Trend 1: Hop-by-hop routing is the most popularly used 

route computation technique, whereas overlay routing 


is the least used (amongst those papers that we have 
surveyed). 

• Trend 2: Considering the hop-by-hop route computation 
technique, we note that this multipath routing technique is 
especially popular in WSNs (compared to other network 
configurations). This is in contrast to other network types 
such as wired networks in which hop-by-hop routing and 
source routing are used a similar number of times. Hop- 
by-hop routing is, on the other hand, comfortably the 
most popular choice for WSNs. 

• Trend 3: Multiple paths can be used in both concurrent/ 
backup configurations. It is observed (based on the works 
presented in this study) that wired/ general networks pre¬ 
fer backup and concurrent multiple paths almost equally, 
while in wireless scenarios, we see that the concurrent 
transmission option is adopted more often. 

A multipath algorithm designer can consult Figures [^ and 
to quickly learn which design choices have traditionally been 
most popular for the particular networking type or configura¬ 
tion being considered. As an example, if we are to design 
a multipathing method for WSNs, then the most popular 
choice is to adopt hop-by-hop routing with concurrent use of 
the multiple paths while focusing on the energy conservation 
metric. 

III. How TO Use Multiple Paths? 

A network-layer multipath framework subsumes both the 
control plane operations of multipathing (that involves con¬ 
struction of multiple paths) as well as the data plane operations 
of multipathing (that involves the use of the constmcted 
multipath paths for forwarding the fiow). After having spent a 
lot of lime focusing on how routes are computed, we now turn 
to the (arguably more) important question of how to use these 
constructed paths. We note here that route computation task is 
typically considered a network-layer function, while the usage 
of resources—and in particular, congestion control through 
which end-user’s transmitting rate is controlled to ensure that 
the overall network performance is not deteriorated due to 



















TABLE V: Summary of Flow Splitting/ Load Balancing Techniques 
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Avoids oscillations which is inherent to load-sensitive routing. It is an online intffic engineering technique bawd on Wrtnirvp routing policies |17DJ. Since 
it ii not restricted to MPLS-based solutions, it is a more general solution compared to TeXCP and MATE. 

Spills Q single TCP flow across multiple paths without resulting in pocket disordering. 

Provides a centralized load-aware scheduler for datacenters. 

Uses estimates of loss rate to balance congestion by auiomuticrally splitting traffic. 

Exploits short-term, partial predictability of traffic matrix in order to adapt to traffic variations It is a load-aware centralized TE solution designed for 
dasaceniers. 

Shifts traffic from one path to another hence, it is a load balancing scheme for end hosts. 

Performs muliipaib sciieduling of mobile users who access heterogeneous wireless networks using muluple interfaces. 
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1998 Preeenu and onalyzex two clauses of rate control algorithms to set prices of titJized resources fairly one algorithm from the perspective of user and another 

one from the network. 

2001 Shows thut shonest-piith routing with positive link weights cun be used to support arbitrary flow splitting to provide an arbitrary set of loop-free routes. 

This Iniemei TE problem is considered without using overlay networks. 

2005 Presents and analyzes a quick load sharing/balancing approach across muluple pailis between a source-destination node pair bawd on end-to-end 
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2006 Presents and analyzes the concept of multipath TCP where cungesuon control and mulupaib routing ore studied jtnntly. Using overlay rouier.s, which 
support source routing, multiputhing In networks is enabled 


Theurelkttl Analysis 

Mitltrpath algorithms for optimal lornear-opUmall decistuns on both routing path andflow splliiing 


Banner ei ol. 181 


2007 


Provides optimal and polynomial-time aJgonthmic soluuons to congestion control using K~palh routing f AlPff) and resiricied midiipalh iRMPl problems, 
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network congestion—is con.sidered a transport-layer function. 
The routing task is managed at the network layer through the 
interaction of routers while congestion control is managed at 
the tran.sport layer at end hosts. These systems unfortunately 
work largely independent of each other resulting in an unwel¬ 
come tussle that arguably is holding back multipathing from 
achieving the immense benefits it can offer p72|. 


We note here that while the layered model is a good 
tool for conceptualizing networking functionality, many im¬ 
portant aspects of the Internet—issues such as reliability, 
flow control, congestion control, QoS, and indeed resource 
pooling through multipath—cannot be pigenholed into a single 
layer. The rigid bifurcation of multipath-related functionality 
into layers results in an antagonistic tussle in which each 
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layer attempts to balance traffic and perform resource pooling 
myopically according to its own needs to the detriment of 
the overall performance. Following the maxim "layering is 
a good servani, hut a had master", we recognize that the 
overall success of multipathing requires solutions beyond those 
that are implemented at the network layer. In this paper, 
therefore, despite our focus on network-layer multipathing, 
we also describe related issues (such as fiow splitting, load 
balancing, and multipath congestion control methods) that are 
meta-network-layer and typically considered in the territory of 
the transport layer. 

In this section, we see how the multiple paths constructed 
by the routing schemes described in Section can be used for 
forwarding the packets. We will see next how the processes 
of flow splitting and traffic engineering together contribute to 
the act of load balancing that is crucial to reaping the benefits 
of resource pooling with multipath. 

Considering first a single flow, we see that the packets of 
a single flow can either use multiple paths by splitting the 
traffic to concurrently use the diverse paths, or alternatively, 
the multiple paths may be used in a backup configuration 
where the primary path is used until a node/ link failure along 
the primary path upon which the routing defaults to the backup 
paths. The purpose of flow splirting in concurrent configuration 
is to achieve the benefits of resource pooling using which a 
collection of multiple paths appears as a single virtual pooled 
link. An important question that arises is how to decide the 
ratio of flow splitting since both of the computed multiple 
paths may have very different properties (in terms of delay, 
bandwidth, congestion, etc.}. There are various requirements 
that we expect from a reliable flow-splitting multipathing 
system: there should not be a need for per-flow state to ensure 
scalability; the system should be tunable, dynamic, and should 
support heterogeneity in bandwidths/ delay. In addition, the 
reordering of packets should be minimized (for which it is 
important to recognize distinct flows, and have mechanisms 
to mark packets as belonging to a particular flow that uses 
a certain sub-path). Coming now to multiple flows, and how 
their packets share the multiple paths available over the net¬ 
work, it becomes important to consider both the utilization of 
the network resources and the fairness of resource allocation. 
This is the prerogative of traffic engineering which is typically 
handled inside the network by ISPs. 

There are five important questions relating to how the 
multiple paths are used that influence the organization of this 
section. 

Firsthy how many of the available multiple paths should be 
used for transmitting the traffic flow? While higher number of 
paths can improve the performance, the management cost also 
increases with increased number of paths. 

Secondly, an important question is whether the multiple 
paths should be used concurrently or in a backup config¬ 
uration? Traditionally most techniques have used multiple 
paths in backup mode for its simplicity although increasingly 
concurrent usage is being adopted to improve performance. 

Thirdly, should the flow splitting/ load balancing be static 
or dynamic? Since the congestion on different paths vary 
dynamically, the load balancing and flow splitting mechanisms 


should also be ideally dynamic. However, in practice static 
flow splitting/ load balancing schemes are also used due to 
their simplicity and low overhead. 

Fourthly, how can we use multiple paths to perform load 
balancing through the ditfusion and control of congestion? In 
particular, traditional congestion control schemes need to be 
adapted for use in multipath scenarios. 

Finally, an important aspect of load balancing is who gets to 
control the process? and where ? Traditionally, there has been a 
tussle between end hosts and the network for network control 
with traffic engineering typically implemented in networks 
while congestion control implemented at the end hosts. 

Apart from discussing these five questions (in the order 
presented) in their respective subsections, we will also present 
a subsection that summarizes the trends presented in this 
section. 

A. How Many Paths to Use? 

While multipathing can provide significant performance and 
reliability benefits, the compulation and maintenance of mul¬ 
tiple paths also entails costs. It has been shown that utilizing 
all the available paths provides only limited performance gains 
over an approach that utilizes a limited subset of all paths Q. 
In fact studies have shown that, to obtain performance gain 
offered by multiple paths, one or two alternative paths, in 
addition to the traditional single path, can suffice Q, fll82| 
with multiple reservation algorithms potentially providing a 
gain in terms of connection establishment time which is 
attractive for interactive applications |182| . 

In previous research, it has been shown through multiple 
studies that the bulk of benefits of multipathing can be 
reaped with only two paths, and that increasing the number 
of routes provide diminishing returns Q. This finding is 
formally established in a widely cited paper by Mitzenmacher 
on randomized load balancing |183| . In another work, Akella 
et al. have also posited similar insights in an empirical study 
focusing on multihoming. More recently, Merindol et al. 
have proposed a multipath routing technique that also relies 
on two precomputed paths. 

Key and Tows ley |184| , |185| have similarly shown that not 
many paths are needed to reap the benefits of multipath. The 
chosen set of paths need to however be dynamically monitored 
for their quality to ensure that the best paths are chosen and 
retained while any poorly performing paths are flushed out 
and replaced. While the dynamic load balancing is done at a 
rapid time scale, the exploration of routes can take place at 
a slower rate. To reiterate, the network-layer functionality of 
computing routes and finding connectivity is done relatively 
slowly while the dynamic load balancing over the chosen paths 
is done through the end systems via the transport layer rapidly. 

It must be pointed out the flow size is an important factor in 
the choice of how many paths to use, and in deciding how to 
prefer dilferent access technologies in multi-homed systems. 
For instance, it has been shown in a study conducted by Deng 
et al. |186| that for short flows, there is no significant benefit 
of using multipath, and selecting the right access network 
technology is more crucial for high performance. 
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B. Concurrent or Backup Paths 

In single-path routings a router only maintains a single route 
to each destination, whereas in midtipaih routings multiple 
routes are maintained. While most common configurations in 
multipath routing are to maintain a single primary path along 
with alternative paths, it is also possible to use multiple routes 
in a backup or concurrent fashion. 

Both of these approaches have their own benefits. Depend¬ 
ing upon a particular network application or a specific metric 
(e.g., reliability or load balancing), a routing algorithm adopts 
any one of tliese approaches. To ensure reliability alongside 
load balancing in a network, the concurrent multipath approach 
P 1 1| , |149| gives the desired results. On the other hand, a set 
of multiple backup paths are precomputed in path failover 
strategic algorithms |187| , which are provisioned in order of 
a specific path ranking mechanism, in the case of failure of 
the primary path (55| , Information about the concurrency 
of multiple paths is summarized for various routing protocols 
in Tables [II] and [nD 

Next, we discuss various aspects of backup and concurrent 
multipathing techniques in more detail in the following sub¬ 
sections. 

/) Backup Multipathing: Routing protocols that make use 
of multipathing with multiple backup paths usually construct 
a primary shortest path in a source-destination pair along with 
a set of alternative paths. The alternative paths in the set are 
maintained and updated according to the network topology 
changes. In case when the primary path fails, then the traffic 
is quickly shifted to one of the backup paths from the set of 
precomputed alternative paths. In this sense, the time incurred 
in route rediscovery is avoided, thus contributing to more 
reliable, fault-tolerant, and resilient routing. 

Legacy protocols, particularly AODV, are improved to sup¬ 
port multiple backup paths in AOMDV |^. Similarly, DSR, 
which is another well-known routing protocol is enhanced in 
ij^ to improve the reliability of routing. In [ |68| |, multiple 
paths are provisioned in an adaptive manner by updating 
AODV to increase packet delivery ratio in networks. Static- 
faii-over routing (SFR) is another important protocol making 
use of alternative backup paths to circumvent network failures. 

Another work )187| discusses the issue of reliability in 
multipath routing in the terms of backup paths. In this work, 
SFR is proposed to study the positive and negative aspects of 
legacy fail-over routing techniques. 

2) Concurrent Multipathing: With modern networking ap¬ 
plications having huge data traffic demands, network con¬ 
gestion has become a significant problem. In legacy single¬ 
path routing protocols, many potential paths in a source- 
destination pair are ignored. In concurrent multipathing, alter¬ 
native paths are exploited to relieve network congestion while 
improving network performance, efficiency, and reliability. A 
large number of works employing multiple paths, along with 
intelligent load balancing techniques, have been proposed in 
the literature. We present a representative sample of these 
works (many of which are based on concurrent multipathing) 
in Tables [H] and [m] 


We note here that utilizing multiple paths in parallel is not a 
recent idea. This general idea has been proposed under various 
names in the literature, viz. traffic dispersion, striping, and 
inverse multiplexing. Traffic dispersion has a rich history in 
networking with Maxemchuk first proposing a traffic dispers¬ 
ing framework known as dispersity routing in 1975 (5^ . This 
approach advocates spreading the traffic spatially (by dividing 
the original message into sub-messages and then transmitting 
them in parallel over disjoint paths of approximately equal 
length) to achieve load balancing and fault handling. The use 
of disjoint paths ensures that transmission errors on different 
paths are statistically independent. Striping pools multiple 
resources to meet capacity requirements even if individual 
links/ network interfaces, or hardware components, cannot 
satisfy the required capacity |188| . Inverse multiplexing refers 
to the phenomenon of aggregating channels to allow greater 
flexibility in selecting bandwidth [189| . 

The concept of concurrent multipath transfer is presented in 
the work |i34| . This concept implies that a source-destination 
pair that uses multiple paths in a multihomed environment can 
reap the benefits in terms of better network utilization and data 
transfer resiliency. This work addresses the issue of shared 
bortlenecks in the computation of multiple concurrent paths. 
A metric based on path correlation is proposed to resolve this 
issue. This metric tells us about the correlation of two paths 
in terms of shared bottlenecks. Lower value of this correlation 
metric indicates that the paths become more uncorrelated or 
disjoint. 

Another important load balancing teclinique that is popular 
at the link layer in recent multipath-based topologies for DCs 
is Valiant Load Balancing (VLB) flT), J]46|, |147| (covered 
in Section lII-Clb| in more detail). This technique has inspired 
many other multipath routing works to cater to the problem of 
congestion and inefficient utilization of network resources. An 
example of VLB-based work that uses uniform fiow splitting 
is presented in |149| in which a design of network backbone 
is provided to provision multiple concurrent paths. 

Recently, concurrent multipathing has become very popular 
and mainstream with the development of multipath transport 
protocols such as MP-TCR The great promise of concurrent 
multipath transmission using transport protocols is the promise 
of true resource pooling using coupled congestion control 
p90| with which it will be possible to shift traffic away 
from congested paths. Recent fluid-flow models p79| , p80| 
have shown the benefit of shifting traffic from congested to 
uncongested paths. In particular, it wa.s shown that multipath 
congestion control can perform the task of load-dependent 
routing stably at shorter time scales (at the same scale of 
TCP congestion control that operates at the order of RTFs). In 
effect, multipath congestion control assumes a role typically 
associated with network-layer routing (of moving traffic to 
another path thereby avoiding congestion hotspots). This mo¬ 
tivates the discussion of multipath congestion control in our 
paper whose focus is on network-layer multipathing solutions. 
In a work related to concurrent transport-layer multipathing, 
Wischik et al. have proposed the principle of equipoise—or 
equally balanced concurrent multipath transport—as a concur¬ 
rent multipathing technique that allows better resource pool- 
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ing, resilience to faults on any subpath, and better handling of 
traffic surges m- 

C. Load Balancing Approaches for Multipath 

Broadly speaking, load balancing and flow splitting ap¬ 
proaches can be either static/quasi-static or dynamic/load- 
aware. The static/quasi-stalic methods are suited for network 
traffic matrices that do not change rapidly, while the dynamic/ 
load-aware methods are suited for changing network traffic 
matrices. There are also a few theoretical works that describe 
optimal, or near-optimal, methods for load balancing and fiow 
splitting. We have summarized major load balancing works, 
with their references, in Table [V] 

I) Static/ Quasi-Static Load Balancing: This approach 
is adopted when traffic matrices are relatively stable with 
changes occurring after a significantly long period of time. 
Flow splitting techniques for load balancing in such networks 
are managed in a static or offline manner. Most of the static 
load balancing approaches aim to fairly schedule the flows 
giving equal preference to Individual links p56| . 

a) Equal Cost Multipathing (ECMP): ECMP gg. [ [T?7| , 
p92| is a multipath routing mechanism that has been em¬ 
ployed in many routing protocols like OSPF and ISIS [ [59) , 
p58| . Intuitively speaking, ECMP can be seen as a simple 
“bonding ' technique that can load balance over equal-cost 
paths. A router can implement three main techniques while 
choosing a next-hop node to forward the traffic including the 
following. 

• Hashing: In this technique, a hash function is applied 
to the header of each packet. The output of this hash 
function detennines over which path this packet is for¬ 
warded to. In (59) , hashing is proposed for the purpose 
of network traffic load balancing over multiple equal-cost 
paths. This work makes use of a 16-bit CRC over the 
five tuples of headers of incoming packets. Five direct 
and one table-based hashing methods are presented. This 
work also presents a solution for keeping the packets of 
a TCP flow in order when it is split over multiple paths 
because of hashing. 

• Per-Packet Round Robin: Fair queuing methods are de¬ 
ployed on a per-packet fashion to provide network re¬ 
sources to multiple contending queues of data packets in 
an efficient manner. In |I56| , efficient and fast running 
algorithms for fair queuing, in which each flow that 
traverses across the network gets equal share of network 
resources, such as weighted round robin, are proposed. 
These can be utilized to load balance the incoming 
contending flows over the equal-cost multiple paths. 

• Destination Prefix-based Approach: In this technique, 
dilferent paths are attributed with different prefixes of 
a destination. The load is then split according to these 
prefixes (59) . 

Since ECMP deals with 'equal-cost' multipathing, it does 
not consider suboptimal routes although they can be potential 
candidates. Constrained Shortest Path First Protocol (CSPF) 
|jl93|, which is an extended shortest path protocol, calculates 


path that fulfills a certain set of constraints, such as minimum 
bandwidth, maximum number of hops, and maximum end-to- 
end delay requirements. 

b) Valiant Load balancing (VLB): VLB is an efficient 
parallel communication technique first presented by Valiant 
| [T7) , |146| , |147[ for general-purpose computer processors. 
VLB is an attractive form of load balacing since it can be 
applied on a per-flow or on a per-packet fashion, and can help 
build networks that can efficiently and reliably support any 
traffic matrix. In the case of VLB, it is sufficient to know 
the capacity of the ingress router of a network instead of the 
knowledge of the whole network traffic matrix. 

In VLB, each data flow is routed towards its ultimate 
destination in two stages: in the first stage, the data flow is 
routed to a randomly chosen intermediate router without the 
consideration of an underlined traffic matrix; the intermediate 
router routes the data flow to its ultimate destination in the 
second stage. 

We explain the concept of the two-staged VLB operation 
with the help of a diagram shown in Figure [^ Consider a 
network of N nodes each with a capacity of /V as shown in 
Figure [^ The two stages work in harmony as follows: 

• Stage I: In the first stage, the incoming traffic that flows 
through the ingress point is divided evenly, regardless 
of the eventual destination, among all the points In the 
network. As the capacity of the ingress point is 7?, and 
if it receives the traffic at this same rate, then the traffic 
is divided among the links giving 

• Stage 2: In the second stage, each intennediary, which has 
received the traffic in the first stage, forwards the traffic 
towards the actual destination. As in the first stage, each 
node receives ^ amount of traffic; so in this stage, each 
link is loaded with ^ traffic by each node. 

The combination of these two stages thus leads to the link 
capacity of ^ each for the normal/ efficient operation of VLB. 
Here, it can be noted that, if we only know the capacity of the 
ingress point (assuming the capacities of all the other points 
are equal), then there is enough information to perform load 
balancing, in a congestion-free way, on an incoming traffic 
load. However, an ingress router must not be overloaded. 
Uniform VLB—where incoming traffic is split equally at the 
ingress point over all the outgoing links—can promise 100% 
network throughput. 

As an example application of VLB, Zhang et al. have 
used VLB to design a backbone network with fault tolerance 
p49| . It has been discussed that, by 10% over-provisioning 
of network elements in tenns of nodes, a 50-node network 
continues to operate in a congestion-free manner with up to 
five node or link failures when VLB is applied to balance 
traffic load in such a network. Over-provisioning can be 
calculated by the ratio of an arbitrary number of failures to 
the total network nodes. 

c) Other Static Approaches: MPLS with optimized path 
(MPLS-OMP) )9n) is used for load balancing purpose for 
label-switched paths. TE. which is further explained in Section 
jlll-Elj is used to balance traffic load across the label-switched 
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paths. In the work ofFortz et al. |158| . OSPF/ ISIS is deployed 
with known traffic matrix to perform TE in a near-optimal 
fashion under the constraints of shortest-patli routing and even 
split of flows over equal-cost multiple links. In the follow¬ 
up works for changing network traffic matrix, Fortz et al. in 
p59| , p6Q| introduced a family of algorithms that adjust the 
link weights according to the changes in the traffic matrix. 
Here, OSPF/ ISIS is used for the traffic engineering purpose 
again. The fact that link weights are altered according to the 
changes in the traffic matrix makes the works quasi-static in 
p59| . |16Q| . In these works, the changes in the network traffic 
matrix are predicted for an intradomain routing. In another 
work |161| , a TE solution for OSPF/ ISIS shortest-path routing 
is deployed for flow splitting purposes. The proposed solution 
computes approximately optimal link loads that provide a 
provable performance bound. In |162| - pS^ , improvements 
for OSPF are presented so that non-shortest paths can be 
included for flow splitting. However, it is observed that a 
delay penalty that increases exponentially is incurred with the 
introduction of non-shortest paths. 

2} Dynamic {Load-aware} Load Balancing: In this ap¬ 
proach the TE solutions split the input traffic in an adaptive or 
online manner |165| — (TST) , |176| . The flow split is governed 
by the changes occurring in the network traffic matrix. We note 
that most of tlie works p65| , |166| , |168| , which adopt this 
approach, address tlie problem of congestion control alongside 
balancing the input traffic load in a real-time manner. 

Load-sensitive routing is prone to collisions. A general 
solution called REPLEX is proposed in p69| to dynamically 
balance the traffic load and also avoid collisions. Another issue 
that can arise when the traffic load is split over multiple paths 
is the disordering of packets of a TCP flow. FLARE p71| 
is proposed with real-time ability to split a TCP flow that 
does not result in disordering of TCP flow packets. MP-TCP 
[ |173| is another online flow distribution method that renders an 
end user to shift the load from one path to another according 
to the input load demands and congestion feedback from the 
network. HALO is proposed in |175| as the first optimal 
LS intradomain routing protocol. It also deploys TE solution 
for load-aware traffic load balancing. Adaptive load balancing 
algorithm for multipath scheduling (ALBAM) is proposed for 
heterogeneous wireless access technologies for mobile users. 


DCs are an important application area in which multipath 
routing with efficient load balancing mechanism provides 
efficient and data intensive operation. Centralized control 
solutions, which manage and schedule the DC traffic flows, 
are presented in tlie literature. Hedera, which is a central 
load-aware scheduler, is proposed in for the DCs. This 
solution schedules flows in a dynamic and adaptive manner 
to efficiently utilize network resources. Another work for 
the same purpose, which is called MicroTE, is presented in 
[ |173| . This technique is also load-aware and it uses short¬ 
term traffic matrix predictions to distribute the network traffic 
over multiple paths. Here, the short-term traffic, which has 
been shown to exist in DCs in this work^ is used to infer 
unpredictable traffic in order to ameliorate congestion. 

In |I72| , a mutual is tic resource pooling architecture named 
PREFLEX was proposed in which the hosts and edge routes 
worked in unison to collectively share the burden, and con¬ 
versely reap the accrued rewards, of balancing traffic over 
multiple paths. PREFLEX is called a mutualistic framework 
since it allows congestion control (performed at the transport 
layer at end hosts) and traffic engineering (performed inside 
the network) to coexist and evolve independently by exposing 
the apparent network preferences and the transport expecta¬ 
tions. 

As we shall see, transport-layer solutions to load balancing 
are increa.singly becoming more popular |I84| , |I9I| , p94| , 
p95| . Multipath transport protocols like MP-TCP already 
incorporate congestion control fealure.s that can automatically 
shift traffic from congested paths onto uncongested paths 
p90| . As we discuss next, in many aspects transport-layer 
load balancing is very appealing. 

3) Theoretical Results of Optimal Flow Splitting: 

“Somewhere ages and ages hence: 

Two roads diverged in a wood, and I — 

I took the one less traveled by. 

And that has made all the difference."—Robert Frost 

It has been shown in multipath routing literature p79| that, 
consistent with Frostian wisdom, taking the less-travelled-by 
path does indeed make all the difference in the context of 
multipath routing in communication networks. In multipath 
terminology, this means that multipath-capable flows should 
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be designed to shift all their traffic from congested paths 
to uncongested paths. Interestingly, it has been shown that 
this load shifting can be done stably by the end hosts at the 
transport layer at a rapid time scale (at the order of RTFs) 
[ |179| . In this section, we consider theoretical works that have 
analyzed congestion control and the benefits of flow splitting 
on multiple paths using fluid fiow models. 

In optimal flow splitting, we are provided a set of paths, and 
we have to decide on what is the most fair and effective way to 
split traffic on the given set of paths. Fairness is an important 
goal to strive for and it entails providing every flow a “fair” 
share of the available bandwidth whenever there is congestion 
at a bottleneck. The most common form of fairness has been 
max-min fairness in which all the connections get the same 
share of the bottleneck. Another common form is proportional 
fairness in which any change in the distribution of rates will 
result in the sum of the proportional rates being negative. 
Tlie rate control of TCP—which is based on additive increase 
and multiplicative decrease (AIMD)—is proportionally fair. 
In weighted proportional fairness, on the other hand, each 
connection is associated with a price and the amount paid 
per rate (and not the rates) is made fair. As an example, a 
connection with a price of two will get the same rate as two 
connections with a price of one each. 

It has been known through the work of Kelly et al. p77| 
that, in a network in which each user chooses a price that 
maximizes its utility, the system evolves to a state that 
maximizes the total network utility. Two classes of control 
algorithms—one from the user’s perspective and another one 
from the network’s perspective—are presented to set prices for 
the utilized resources in a fair manner. 

In a follow-up work. Kelly et al. |179| considered the 
important problem of joint routing and rate control where 
it was demonstrated that load sharing among multiple paths, 
in a multipath routing scenario, can be achieved in a similar 
time scale in which rate control is performed. But this quick 
load sharing approach must be performed while keeping in 
view the network stability (responsivene.ss vs. stability). This 
is why, in this work, a method is proposed to analyze end-to- 
end stability in the joint routing and rate control scenario with 
the presentation of a sufficient condition for this purpose. This 
sufficient condition works in a distributed manner and deals 
with the stability of end-to-end algorithm for joint routing and 
rate control. 

Han et al. have also considered the problem of joint multi- 
path routing and congestion control with multipath-TCP p8Q| . 
This problem is studied based on an overlay network of routers 
that .support source routing. Wang et al. in |178| have also 
considered arbitrary traffic split among multiple paths in a 
shortest-path routing scenario. But here, an overlay network 
is not con.sidered in this TE approach. Links with positive 
weights have been used to support the distribution of arbitrary 
input traffic matrix. 


D. Multipath Congestion Control 


practice, congestion on the Internet Ls passively controlled 
at the end hosts through TCP that observes and responds 
to packet losses. Alternatively, the network nodes can also 
actively manage their queues using active queue management 
(AQM) technology—using algorithms such as random early 
detection (RED) |196|. explicit congestion notification (ECN) 
p97| , or controlled delay (CoDel) |198| — to take preventive 
measures to avoid congestion. 

Tlie purpose of multipathing is to enable resource pooling. 
For effective resource pooling, it is important to develop the 
abstraction of a single-pooled resource from the collection of 
diverse resources. Tlie key to the creation of this abstraction is 
effective load balancing. More relevant to multipathing, load 
balancing can be performed over multiple resources (e.g.. the 
load is balanced over multiple links/ paths). With multipath 
resource pooling, the aim is to pool multiple paths and to 
provide the illusion of a single much more powerful path. This 
resource pooling can provide the benefits of higher throughput, 
improved fault tolerance, and the capacity to handle traffic 
surges. 

There is a lot of ongoing work in the research community 
in developing efficient multipath congestion control p99| , 
p85| . [ |2QQ| . This interest is further buoyed by the ready 
availability of path redundancy and diversity in modem net¬ 
working environments such as datacenter networking. For 
maximum efficiency, there should be a .strong emphasis on the 
construction of holi.stic multipathing frameworks that jointly 
optimize the various facets of network-layer multipathing. 


7) Coordinated or Uncoordinated Control: With multipath 
congestion control, an important consideration is whether 
to use uncoordinated or coordinated control. Uncoordinated 
control can be implemented in the current Internet (running 
unmodified TCP) in a straightforward fashion. This makes 
uncoordinated control attractive for application-layer solutions, 
and explains the use of such schemes by P2P frameworks. 
With coordinated control, on tlie other hand, the coordinated 
controller can better balance load when the paths are fixed. 
Kelly and Voice |179| and Key et al. 1 184| have studied con- 
ge.slion management in multi-flow environments. They have 
shown that congestion response for each subflow should be 
coupled to ensure stable and efficient rate allocation between 
the subflows. This i.s important both when a path fails and also 
when a new path appears. This has been .studied using tools 
of game theory by Key and Towsley |I85| by the formulation 
of a path selection game which studies the choices a user 
should make about the set of throughput-optimal paths to use 
seeking to maximize their own net utilities. The ensuing Nash 
equilibria i.s shown to be welfare-maximizing social optima 
for the case of coordinator controller and for the restricted 
case of uncoordinated controller that do not exhibit RTT bias 
(unlike TCP which does). For uncoordinated controllers with 
RTT bias, the Nash equilibria may not coincide with the 
social welfare-maximizing optimal. Coordinated controllers 
have also been shown in p85| to improve upon greedy least- 
loaded resource selection as proposed by Mitzenmacher. 


Apart from the issues of route computation and traffic 
engineering, another important issue is congestion control. In 


2) Congestion Exposure: Today’s traffic management pro¬ 
tocols are typically suboptimal since they disregard (i) protocol 
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Interactions between congestion control, routing, and traffic 
engineering; and (ii) the tussle that emerges from independent 
network or end-host control of congestion. The task of har¬ 
monizing congestion control, routing, and traffic engineering 
is challenging due to the different timescales they act on 
(short, medium, and long timescales respectively}. In recent 
times, there is renewed interest in integrating routing and 
congestion control in a cost-effective fashion. The EU-funded 
Trilogy project |201| — named as tribute to the crucial trilogy 
of *'routing, congestion control and cost effectiveness"—aimed 
at improving the quality of the internal functioning of the 
Internet through cost-effective resource pooling and integration 
of the routing and congestion control functionalities. 

An important component of the Trilogy architecture aim¬ 
ing to plug a gap in the Internet's architecture is Re-ECN 
(re-feedback of explicit congestion notification)—which is 
an economics-based congestion pricing resource management 
framework that holds users/ applications for the congestion 
they create. It has been shown in previous work that resource 
allocation through the traditional TCP method can be gamed/ 
exploited (e.g., a user can open multiple TCP flows to gain 
disproportionately high throughput at a bottleneck link) since 
the rate control is performed at the end hosts. The network 
operators have limited visibility into the congestion in the 
network and their focus has been traditionally on managing 
utilization. The traditional focus on ‘TCP-friendliness" fl2Q2| 
and the focus on equal fiow rates have been challenged re¬ 
cently and their fairness and desirability have been questioned 
| | 20 ? 1 . 

An important protocol in the Re-ECN framework is the 
Congestion Exposure (ConEx) protocol that can also lead to 
better integration of end-host applications and service provider, 
and harmonious interworking of various stakeholders on the 
Internet |2Q3| . ConEx attempts to fairly “divide the resource 
pie" and mitigates the indiscriminate abuse of the Internet 
by an atypical form of traffic policing that only pushes back 
users who are limiting other users' freedom. This is done by 
charging users according to the congestion they cause, rather 
than by the volume of traffic generated. The underlying idea 
here is that we should not limit the traffic volume generated 
by users unless the resource share of other users is being 
impacted. Thus, if there are more people who wish to send 
data, there is a higher price of transmission, and vice versa. 
Without such a system, the system can be exploited causing 
indiscriminate use of the Internet leading to tussles between 
ISPs and user applications such as P2P |203| . 

In ConEx, the packets carry an accurate prediction of the 
congestion they expect to cause downstream, which is visible 
to the ISPs/ network operators who can now act on the 
congestion signal being exposed to them by the end hosts 
p04| . This information can then be used for a variety of 
purposes including congestion policy, traffic engineering, and 
accountability. ConEx encourages well-behaved applications 
that aid load balancing (e.g., by delaying the transmission of 
traffic from congested times to uncongested times) and can 
help facilitate better network economic pricing models. This 
can allow ISPs to charge users for congestion caused while 
also providing users incentives to fairly consume resources 


since they have to now pay for the congestion they are causing. 
There is also the scope of novel tussle-aware congestion 
management schemes that utilize information made available 
to policing of network congestion |2Q5| . 

ij New Developments in Congestion Control: As an inter¬ 
esting example of a new congestion control algorithm, that 
involve greater interaction between the network and the end 
systems, consider Low Extra Delay Background Transport 
(LEDBAT)—a protocol used by Apple and BitTorrent for large 
background transfers that account for up to an approximate 
15 to 20% of the the global Internet traffic—which works 
by limiting congestion induced by the LEDBAT flow itself in 
the network. There has also been work in machine-learning- 
based congestion algorithms that are customized for user and 
network preferences. In the ReMy work, proposed by WInstein 
et al. [ |206| , the computer-designed endpoint algorithms meet 
the defined preferences and assumptions about the users, 
application traffic, and network much belter than human- 
designed algorithms (even those algorithms that deploy code 
inside the network). In other works, ReMy searches for the 
best congestion-control algorithm and optimizes the expected 
objective over prior assumptions. Tliis work promises that 
we can retain the end-to-end design philosophy and work 
with a simple “dumb" network, and have a computer-designed 
algorithm operates at end points and outperforms even the best 
in-network algorithm. 

E. Which Layer/ Where to Perform Load Balancing/ Traffic 
Engineering? 

For the success of a load balancing/ traffic engineering 
scheme on the Internet, it is important that the proposed frame¬ 
work holistically integrates the various layers of the Internet 
architecture. The network alone is incapable of managing 
the traffic since the routers do not have enough information 
about the end-to-end traffic fiow and thus cannot make an 
information decision about the paths to be used. The end host, 
on the other hand, does not know about the network traffic and 
cannot ensure fairness of the network alone. The problem with 
performing congestion control at the transport layer is that the 
control process may not have enough time to kick in for small 
flows. In many cases, application-layer mullipathing is best 
suited to serve the unique needs of the application and can 
be used to choose from which server to download (as is done 
in P2P applications such as BitTorrent that already inherently 
utilize multipath). 

Our discussion on transport-layer mullipathing in this pa¬ 
per, despite our network-layer focus, is motivated from the 
convergence of the research community on the important 
role that congestion-controlled multipath transport will play 
in the future multipath-enabled Internet. It has been argued 
in literature, following the work of Kelly and Voice p79| , 
that end systems using transport-layer functionality, rather than 
the networking nodes implementing network-layer protocols, 
are best placed to implement load balancing—particularly 
across network domains—on the Internet. The information 
below the transport layer is too coarse to dynamically control 
the traffic and can only enable coarse-grained functionality 
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(such as shifting of a few elephant flows away from hotspots 
within a network domain). Indeed, many application-layer 
protocols (especially P2P protocols such as BitTorrent) already 
perform multipath in their function. Implementing a generic 
multipath load balancing/ resource pooling functionality can 
help implement a new ‘narrow waist* of the future Internet. 
Tliis also naturally fits with the existing role of TCP in 
implementing congestion and rate control functionalities. 

Broadly speaking, there are three main techniques for 
multipath resource pooling: (i) Routing-based traffic engineer¬ 
ing; (ii) Application-based load balancing between multiple 
servers; and (iii) SDN-based approach to traffic engineering. 

}) Rouler-hased Traffic Engineering {TEj: Since it is com¬ 
mon for large networks to offer significant path diversity, there 
is great interest in using TE to increa.se network efficiency. A 
common approach to implement flexible forwarding is to first 
classify packets (according to application requirements with 
packets for the same flow being similarly classified according 
to the ‘type of service' field in IP packet header) and then 
to map packets to different paths by having an edge router 
examines the packet header, evaluate the path characteristics, 
and determine a suitable path for a certain cla.ss of traffic. 

To forward packets using the alternative paths, routers that 
are part of the communication session establish a logical link 
with each other using tunneling. The tunneling approach can 
be utilized in MPLS or IP-in-IP tunnels. Although an extra 
header is needed for encapsulation; however, compared to 
the much longer prefix match of destination address, a label- 
based look up is an easier and faster approach. Using explicit 
routing, each router can specify a path using a packet header, 
or a label embedded inside the packet header. Tunneling is 
a useful approach for special applications that need end-to- 
end bandwidth guarantee. This approach is also employed in 
a way that a complete router-level path is specified along witli 
IP options. Moreover, each intermediate node performs a label 
lookup to discover the outgoing link. 

Traffic engineering using network-layer routing functional¬ 
ity is often performed at slow time scales and is not automated 
(often with a human in the control loop). Internet traffic 
engineering functionality, like mo.st of the Internet's control 
functionality, has emerged as a melting pot of numerous 
piecemeal fixes (such as the congestion control fix provided by 
Jacobson’s work in TCP; the traffic engineering fix provided 
by MPLS; and the policy routing fix provided by BGP). 

The BGP protocol, developed mainly for advertising policy- 
compliant loop-free connectivity, has limited support for traffic 
engineering. The cumbersome process of establishing peering, 
provisioning links, as well as tuning routing and QoS policies 
makes traditional router-based traffic engineering unattractive. 
This, along with the coarse-grained traffic engineering func¬ 
tionality by protocols such as BGP, has motivated the devel¬ 
opment of other approaches (such as the two approaches— 
application-based and SDN-based, traffic engineering—that 
we discuss next). 

2) Application-based Load Balancing/Traffic Engineering: 
Tlie traffic engineering decision can be made by the applica¬ 
tion itself at the edge host. As an example, P2P applications 


like BitTorrent utilize multipath—but often In a fashion that 
is oblivious and impervious to the ISP (e.g., the peers are 
selected randomly without necessarily looking for peers that 
are local to the same ISP)—thereby impacting the efficiency 
and costs of ISPs. There has been some work done to allow the 
ISPs greater flexibility in managing the P2P applications. For 
instance, the P4P frainework |207| aims to equip the provider 
with the ability to manage P2P applications by performing 
smarter peer selection, better traffic distribution, and higher 
transfer speeds, all at a lower cost to the ISP. P4P does this by 
allowing the ISPs to notify the clients about the preferred local 
clients. Similarly, the Ono system aims to “tame the torrent” 
of cross-ISP data generated by BitTorrent by modifying the 
BitTorrent client to prefer local peers |208| . 

While application-layer load balancing and traffic engi¬ 
neering can be quite effective, it can conflict with what the 
network operators are doing, causing an unfortunate tussle. 
In particular, application-layer multipathing, as performed in 
P2P applications, is suboptimal due to the lack of availability 
of topological information to the application layer at the end 
hosts. The Application-Layer Traffic Optimization (ALTO) 
framework |2Q9| proposes to equip distributed Internet ap¬ 
plications with sufficient topological Information exposure to 
optimize resource utilization. With ALTO, P2P performance 
can be improved by allowing better-than-random peer selection 
allowing P2P traffic to align itself with ISP constraints. 

3) SDN-based Traffic Engineering: In recent times, SDN 
architecture has emerged as a viable platform to facilitate QoS- 
aware and efficient traffic engineering. A popular use case 
of SDN is to use traffic engineering aided by a centralized 
network-optimizing vantage point in datacenters, campus net¬ 
works, and in service provider networks. 

The modem SDN trend—where network policies can be 
disseminated from a centralized controller to network elements 
such as switches—can help fundamentally transform TE as 
demonstrated most dramatically by Google with their traffic 
engineering system B4 (55) . B4 uses an OpenFlow-based SDN 
architecture that achieves nearly 1(K)9 p traffic utilization when 
splitting application flows across multiple paths while meeting 
application priority/ demands. SDN has recently used by 
Me Connie k et al. |210| to provision real-time alpha fairness- 
based TE. The proposed algorithm executes in the range of 
milliseconds and can be used as an online tool. 

An excellent survey for TE in SDN environment by Akyildiz 
et al. is presented in |21 1| . We note that there are four frontiers 
of TE for the modern networking age of SDN. As mentioned 
in |211| , these four frontiers are: (i) flow management, (ii) 
fault/ failure tolerance, (iii) topology/ network policy update, 
and (iv) traffic analysis/ monitoring. This is depicted in Figure 

Here, we briefly describe these frontiers with the relevant 
reference works, which treat these frontiers in the perspective 
of SDNs and DCs. 

• Flow Management: In a SDN environment, each new type 
of flow is routed towards a SDN controller by an ingress 
switch. The controller then decides the path for this and 
the similar prospective flows. The controller disseminates 
this new path information to all the relevant switches. 
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Fig. 5: TE Frontiers for SDN (Source: D21ll). 


However, if there are a large number of new incoming 
flows, then it takes longer for a controller to decide 
new paths for each new category of flows, as well as 
to program the data planes of the relevant switches. This 
increases delay and can create problems, especially for 
the DCs, which have been solved using intelligent and 
Innovative TE solutions. Hedera (5?) is proposed to deal 
with these issues using a global view of active flows in the 
network. It creates non-conflicting paths for the flows as 
per the current routing and traffic demands in a real-time 
manner. Another similar work is Mahout fl2I2| , which 
is an end host-based DC traffic management solution. In 
this study, traffic management is carried out by detecting 
large flows or elephant flows at the bulfers of the end 
hosts. Then, TE is applied to manage these flows using 
multiple paths with low control overhead. 

• Fault/ Failure Tolerance: In a centralized control envi¬ 
ronment of SDN. failure recovery is a challenging task. 
In case of a failure, a controller must provide the affected 
switches or other network elements/ resources with a 
fail-over routing path(s). These paths can be considered 
as backup paths where the traffic is shifted once the 
primary path(s) fail. Provision of such type of alternative 
paths in a real-time fashion can be a challenging task 
for a SDN controller. Two types of failures can occur, 
so two types of failure recovery mechanisms are needed, 
namely (i) data-plane failure recovery and (ii) control- 
plane failure recovery. For data-plane failure recovery 

restoration and protection operations for network 
elements, like nodes and links, must be performed; while 
for control-plane failure recovery [ |214| , a single-point 
failure must be avoided. Due to the dependency on the 
controller in a SDN environment, backup and replication 
techniques for the controller failures are adopted. 

• Topology/ Network Policy Update: Topology update in 
SDNs, unlike conventional networks, is related to the 
dissemination of network policy changes. Tlie control 
plane of a SDN is responsible to plan for the updates 
in the network policy mles. The challenge is maintaining 
the consistency of any updates in the policy rules in the 
data planes of all switches so that they are operating 
under the new rules. Problem arises during the update 


process of the data planes of the underlying switches. 
Active flows, which are still under the influence of earlier 
network policy, can gel affected or completely discarded 
during this update process. This phenomenon degrades 
the QoS of a SDN-based system. So, it is desired to 
perform the topology update mechanism in a near real¬ 
time fashion so that only minimum degradation occurs. 
Network policy update process is a planned mechanism 
by a SDN controller. Per-packet/ per-flow consistent 
and non-conflicting updates are ensured by solutions 
presented in [ |215| , |2I6| . In these works, tlie controller 
requires an ingress switch to label incoming packets or 
flows with a tag to tell the policy imposed on this packet 
so that the switches along a path can process accordingly. 
By this way, a single packet is processed according to 
only one policy—either the new or the earlier policy, but 
not both simultaneously. 


Traffic Analysis/ Monitoring: This frontier can be con¬ 
sidered as a vital part for the whole TE mechanism 
in SDN. Accurate and efficient implementation of this 
frontier renders other frontiers to operate in a systematic 
cohort with high accuracy. In traffic analysis, difterenl im¬ 
portant observations—such as programming bugs, traffic 
data, network stale and traffic/ data patterns—are taken. 
These measurements are important for timely detection 
of malfunctioning or the failures of network elements. 
Unfortunately, most of the current SDN-based systems 
still employ monitoring tools offered by the conventional 
IP domain systems. This approach often incurs huge 
delays, control overhead and inaccuracies in a SDN 
environment. A need for modem and SDN-related traffic 
monitoring tools is felt due to the aforementioned issues. 
Various traffic monitoring tools for SDN environment 
are available both from the industry, e.g, |[55), [|217|, 
and the academia, e.g, ]218| , |219| . A few SDN-based 
traffic analysis/ monitoring works are discussed next as 
examples. MicroTE (54) is a load-aware TE solution that 
predicts short-term traffic matrix that helps in adapting to 
the changing TE patterns. PayLess is another work that 
collects aggregate statistics at packet, flow and port levels 
in a real-time manner with high accuracy and low control 
overhead |22Q| . 


F. Summary of Trends 

In this section, we summarize the major trends in how 
multiple paths are used in wired and wireless networks. 
We refer to Figures and for a visual depiction of the 
design choices made by various multipath protocols proposed 
for wired and wireless networks, respectively. We observe 
that multiple paths can be used in both concurrent/ backup 
configurations. It is observed (based on the works presented 
in tliis study) that wired/ general networks use backup and 
concurrent multiple paths almost equally, while in wireless 
scenarios, we see that the concurrent transmission option is 
adopted more often. Increasingly, the control of flow splitting, 
congestion control, and load balancing is being performed at 
the transport layer and the recent development of multipath 
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iransport protocols can be influential in strengthening this 
trend. With the recent proliferation of DCs, the use of dynamic/ 
load-aware flow splitting and SDN-based traffic engineering is 
also becoming more popular and will grow in importance in 
the future. 

IV. Open and Current Research Issues 
A. Pragmatic Deployable Mullipalhing 

}) More Flexible Support For Policies: As the domain 
of applicability of multipathing expands with the growing 
popularity of multipath, the important problem of harmonizing 
the aim of efficient multipathing with the policies of the var¬ 
ious stakeholders involved becomes more complex. The BGP 
protocol has traditionally provided limited support for traffic 
engineering. In particular, BGP does not natively facilitate 
common tasks such as balancing load across multiple links 
to a peering AS |22I| . Due to the wide deployment of BGP, 
a clean slate approach to interdomain multipath routing that 
breaks away significantly from the normal operation mode of 
BGP is unlikely to be deployed due to the central role of 
BGP on the current Internet. There have been various works 
proposed to introduce flexible interdomain multipath routing 
but the problems are far from fully resolved. 

In addition, mulitpathing, by its very nature, brings to 
bear the possibility of policy complexity and conflict (both 
locally as well as globally). For example, a local conflict 
can arise between the objectives of energy efficiency and 
minimizing price when using WiFi/ Bluetooth/ Zigbee vs. 
Cellular |222| : whereas example of a global conflict could 
be the need to respect the interdomain peering arrangements 
while performing multipathing efficiently |[5^. On a mobile 
device, multipathing brings about interesting tradeoffs: e.g., if 
the desire is to plainly maximize battery life, then the lowest- 
energy interface (Wi-Fi or 3G) will be used; else if the interest 
is in maximizing throughput, then it will make more sense 
to use MP-TCP-like protocol to concurrently utilize both the 
interfaces; finally, if the preference is for price minimization, 
then a host of factors including the applicable pricing (whether 
roaming or not) will have to be incorporated. While there are 
a number of ways in which the end points could interact with 
the routing systems at the intradomain and interdomain levels 
to offset the policy tussle and ensure efficient multipathing, 
nothing concrete has been agreed upon yet, and this remains 
an open research issue. 

2) Seamlessly Handling Heterogeneous Paths: By its na¬ 
ture. multipathing, especially in multi-homed devices/ net¬ 
works, involves traversing heterogeneous technologies/ do¬ 
mains. As an example, Wi-Fi and cellular interfaces are perva¬ 
sively available on modern wireless devices. Wi-Fi and cellular 
have very different characteristics: Wi-Fi typically exhibits 
stable RTTs with relatively high loss rates, while cellular has 
variable and large RTTs with low loss rates (with the large 
variable RTTs attributed to “bufferbloat" |223| , |224| that 
refers to unnecessary provisioning of buffers throughout the 
network which hinders the congestion control process). The 
heterogeneous characteristics of the underlying technologies 


on the Internet can cause performance issues. It has been 
shown that MP-TCP performance can suffer when Wi-Fi and 
cellular flows are coupled due to cellular bufferbloat which 
causes the problem of flow starvation and idle restart p25| . 
More research needs to be done on how to tame the inherent 
heterogeneity on the Internet so that multipathing can show 
its full benefits. 

3) Multipathing and Cluster-based Netivorks: Despite the 
many advantages offered by multipathing, the high cost of 
maintaining multiple paths impedes wide-scale adoption. A 
potential solution to this problem is clustering in which nodes 
are grouped into logical groups/ clusters, with a clusterhead 
elected to represent the group. Clustering has many routing 
applications such as its ability to limit the amount of control 
messages exchanged, and its ability to aid abstraction (e.g.. the 
Nimrod routing architecture |226| uses clusters to abstract the 
internal topology, hiding away the details and only revealing 
additional details upon request). While clustering has been 
widely investigated, there is a lack of research work on 
supporting multipathing in clustered networks—particularly 
the effects of the underlying cluster structure (e.g.. cluster size) 
and clustering operations (e.g., cluster maintenance) on mul¬ 
tipathing. Further investigation can be pursued to investigate 
clustering as an effective solution to improve the scalability 
of multipathing. 

4) Interworking with Middleboxes: Middleboxes are net¬ 
work entities that cause interruptions in end-to-end protocols 
like TCR Middleboxes can be of various types and purposes, 
such as firewalls, network proxies, network address translators 
(NATs), and so on. Tliese entities, unlike routers, are link 
layer entities that check the layer three infonnation of a 
data packet. Tlie proliferation of these middleboxes is one 
of the reasons for the ossification of the current Internet. 
With modem, often software-defined, networks and end hosts 
with potentially multiple interfaces/ addresses, the need for 
an end-to-end multipathing protocol is direly felt. The TCP 
has an inherent capability to support multipathing through the 
correct settings of the option fields in TCP packets. The major 
impediment in the way of realizing this multipath dream is 
middleboxes as they are tamed to work in an ossified fashion of 
processing packets assuming no prospective evolution. While 
making TCP to support multipath, it has to be considered that 
how the space of sequence numbers is to be used |227| , [ |228| . 
Do we need a single space for all the flows or a separate 
space will work just fine? If there are gaps in the sequence 
space of multiple TCP flows, then what might happen at the 
middleboxes? How will transmission, ordering and retrans¬ 
mission work in the presence of traditional middleboxes p27| , 
p28| ? Future research can be pursued to investigate these open 
questions. 

While traditional middleboxes can create problems for mul¬ 
tipath TCP, middleboxes can also come to the aid of MP-TCP. 
For example, Detal et al. have proposed a custom middlebox 
known as MiMBox that works as a protocol converter between 
MP-TCP and TCP |229| . Since MP-TCP requires modification 
of the end hosts, it suffers from the chicken-and-egg deploy¬ 
ment problem; MiMBox can facilitate the transition of the 
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Internet's transport towards MP-TCP. 

B. Addressing Multipathing Holistically 

Tlie Internet’s control architecture is not a result of carefully 
planned organization of functionality. Such an *'architectural 
soup of network controls” is ill equipped to support multi- 
pathing and there is a dire need of a holistic control framework 
for multipath routing. Some particular problems are described 
below. 

1) Cross-layer Multipath Support: Besides the network 
layer, there has been work on exploiting the benefits of mul¬ 
tipathing at other layers; hence it is important to deploy and 
exploit cross-layer optimization in order to develop effective 
Internet-wide multipath solutions. The network layer multipath 
solutions must interwork with application, transport and link 
layers as described below. 

The interaction of the application layer and the network 
layer can help to create ehiclent multipath solutions. The 
development of overlay routing often requires cross-layer 
support. It is often the case that certain routing-related observa¬ 
tions (such as communication failures at the application layer 
in which the network layer fails to detect) can only be .suitably 
made at the end hosts. The end hosts can utilize the overlay 
networking technique at the application layer to implement 
multipath routing on top of today’s relatively inflexible routing 
system. It is also often the case that datacenters use load bal¬ 
ancers to direct flows at the application layer to suitable servers 
keeping in view the network conditions. These observations 
motivate greater support for cross-layer interactions between 
techniques at the network and application layers. 

Previous research has also demonstrated the importance of 
the transport layer in efficient multipathing. Kelly and Voice 
[ |179| have shown that, while the network layer is well suited to 
provide structural information, the load balancing task is best 
managed at the transport layer. In particular, the key constraint 
on the responsiveness of each router is the RTT of that route 
(the information of which is conveniently available at the trans¬ 
port layer). There is also a great need for increa.sed interaction 
between the transport layer and other layers. For example, 
transport layer can work with network-layer multipathing to 
reorder out-of-order packets arriving at the destination. There 
is thus a strong motivation of greater interaction between the 
network and transport layers for efficient multipathing. With 
recent research progress in transport layer protocols (with 
works such as MP-TCP, SCTP, along with numerous cross¬ 
layer solutions |168| being proposed), the intersection of the 
network and transport layers for multipathing is expected to 
become a promising area ripe for further research. 

Finally, exploiting the information at the link layer is also 
very important, especially for wireless networks where channel 
allocation and assignment are performed to encourage diver¬ 
sity |23Q| , |231| . It is noted here that in wireless networks, 
transmissions on orthogonal channels can take place simul¬ 
taneously, and thus orthogonal channels can be considered 
as logically distinct channels/ paths. Multipath protocols can 
increase their efficiency by exploiting the multiplicity of routes 


made available through the use of channel-diversity in multi¬ 
channel wireless networks |232| . In a similar fashion, the 
interface diversity (or the availability of multiple radio network 
interface cards) of multi-radio wireless networks fl233| can be 
exploited to improve the performance of multipath protocols. 

2) Mitigating the Tussle Between the Various Stakeholders: 
Due to the fact that resource pooling mechanisms based on 
multipath are built on multiple layers and levels, and that 
a coherent framework addressing all aspects of multipathing 
(route computation, load balancing, congestion control, etc.) 
is lacking, it is important to ensure that the various resource 
pooling mechanisms work well in harmony and do not con¬ 
flict. Towards this end, there needs to be greater synergy 
between the end systems, the end networks, and intermediate 
autonomous sy.stems so that potential mechanisms/ policy 
conflicts (that can lead to suboptimal performance) can be 
avoided. 

As an example. P2P applications .such as BitTorrent perform 
load balancing at the application layer independent of the ISPs 
by preferentially retrieving data on uncongested paths. Such 
an arrangement could come clo.se to optimizing the cost for 
congestion pricing. However, with AS peering managed by 
the ISPs with arbitrary pricing models, the conflicting policies 
of the end systems and the ISPs can lead to a significant 
performance loss. It has been shown that the ‘cost of anarchy’ 
resulting from the end systems and the ISPs adopting different 
metrics for congestion can potentially be arbitrarily high fl234| , 

mg. 

While network-layer multipathing solutions will play an 
important part in the future multipath-capable Internet, ef¬ 
ficient resource pooling of multiple paths will eventually 
require cross-layer support and a seamlessly integrated re¬ 
source sharing framework. It has been argued convincingly 
that the traditional inability of the Internet to progress beyond 
a single-path paradigm was less rooted in the lack of routing 
solutions but more so in the inflexible resource sharing model 
of the Internet which had an inevitable tussle between the 
hosts and the network |236| . This tussle restricted the overall 
promise of resource pooling since the independent policies 
of the stakeholders often differ—usually at the expense of 
each other. Effectively, the various stakeholders are performing 
load balancing in a siloed fashion (independently and while 
trying to be inconspicuous to others). This situation is far from 
ideal since the interaction between the stakeholders will be at 
best a form of commensalism (a situation in which one entity 
benefits while the other parties remain unaffected) and at worst 
destructive for each other. 

In the EU-funded Trilogy project [2Q1| , there has been 
an explicit attempt to acknowledge and manage the tussles 
between the different stakeholders in the network. Trilogy 
projects propose a new tussle-aware Future Internet architec¬ 
ture that intrinsically incorporates multipath routing and trans¬ 
port while integrating technical solutions with socio-economic 
factors. Trilogy project proposed four general design principles 
for the future Internet |237| , |238| . Firstly, the Infannation 
Exposure Principle requires the exposure of sufficient infor¬ 
mation about resource usage to support efficient and timely 
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Fig. 6: A depiction of the holistic multipathing architecture 
proposed by the Trilogy future Internet architecture. 


resource allocation. Secondly, the Separation of Policy and 
Mechanism Principle proposes to separate higher-level pol¬ 
icy decisions from standardized implementation mechanisms. 
Tliirdly, the Fuzzy Ends Principle proposes a mutual!Stic 
framework in which the endpoints can delegate some functions 
to the network. Finally, the Resource Pooling Principle —most 
relevant to multipathing—dictates that the network resources 
should be pooled effectively so that spare capacity can be used 
effectively to counter overload and faults. 

The Trilogy architecture (shown in Figure]^ serves as a can¬ 
didate embodiment of these principles. The overall architecture 
consists of a packet forwarding layer and a transport services 
layer. The packet forwarding layer comprises two players: 
(i) the reachability plane which is responsible for hop-by- 
hop routing necessary for enabling network-wide reachability, 
and (ii) the resource plane which is responsible for rate 
control and for deciding how to perform resource allocation 
between different packets/ flows. The transport services layer, 
on the other hand, manages the functionality of reliability, flow 
control, and message framing transparently from the packet 
forwarding layer. The transport layer is implemented end-to- 
end and interacts with the application layer. 

Multipath TCP and Multipath Routing, along with Conges¬ 
tion Exposure, comprise the three building blocks of the Tril¬ 
ogy architecture. Multipath TCP can directly exploit multiple 
paths on multi-homed endpoint.s whereas multipath routing 
allows the computation of disjoint path.s even for single-homed 
devices (multipath theory indicates that the benefits of multi- 
path can be reaped even without complete path disjointness 
as long as there is no shared path bottleneck). A synergy 
of transport and network layer multipathing is thus proposed 
in a holistic multipath solution that can allow faster traffic 
engineering (now perfonned end-to-end at the timescale of 
RTTs). Facilitating this interaction is ‘congestion exposure’ 
which enables all nodes to see the congestion they cause by 
sending traffic. Congestion exposure also allows the operator 
to judge which paths are good and should be made visible 
to the end users, while also being able to see if the traffic 
engineering performed by the end hosts is satisfactory and 
not malicious. 


C. Multipathing and Future Internet Atvhirectures 

As explained in Section we anticipate that multipathing 
will be a big part of the future Internet. Many of the future In¬ 
ternet architectures utilize multipathing in their framework. In 
this section, we will describe the role multipathing plays in the 
information-centric networking (ICN) and SDN architectures. 

]) Multipathing in Information-Centric Networks: The ICN 
architecture employs content caching as an intrinsic and ubiq¬ 
uitous part of the networking infrastincture. A key aspect 
of ICN is to decouple location and content thus allowing 
content to be retrieved chunk-by-chunk from any location that 
is convenient. ICN inherently supports multipathing since a 
node's ’Interest' for a content is not sent to a single location 
but can be sent through several interfaces with the replies being 
recomposed for application use. In addition to resource pooling 
of end-to-end resources and paths (as is done by multipath 
TCP protocols), resource pooling can be done on the basis 
of in-network resources (as is proposed in the ICN context 
with the in-network resource pooling work proposed recently 
[ |239| ) to improve reliability, end-to-end throughput, network 
efficiency, and flexibility. 

2) Using SDN for Multipath Provisioning: More research 
needs to be conducted to discover how to best use the 
centralized paradigm offered by SDN, along with its more 
flexible control abstractions, for managing and provisioning 
multipath routing. SDN has been recently used by McCormick 
et al. [ |210| to provide real-time alpha-fairness based traffic 
engineering. The proposed algorithm executes in the range of 
milliseconds and can be used as an online tool. Using the SDN 
architecture to split the control and data planes can also enable 
rapid response to network failures |240| . In particular, in a 
SDN-based .setting, multipath routing can not only improve 
performance and reliability, but can also forestall the need to 
dynamically recompute routes in response to failures. 

D. Deploying Diversity Efficiently 

One of the main benefits of multipathing is the plurali.sm 
and diversity afforded by the presence of independent links. It 
has been observed in a wide variety of settings that systems 
exploiting diversity and pluralism can outperform individual 
systems that do not rely on diversity (17) , |241| . In this regard, 
the following are some important points to con.sider. 

• Quantijying the Available Diversity. Various quantitative 
measures of diversity have been developed in various 
sciences including entropy measures, variation measures, 
distance measures, and attribute-based measures 024I| . 
Although it is unlikely that a single universal “one-size- 
fits-air’ quantitative definition can be defined, there is a 
strong need of quantative measures of multipath diversity 
to build a rigorous theory on multipath routing. Some 
initial works on using diversity index in networking have 
been proposed |242| , |243| , but there is a lot of scope 
for further work. 

• How to Use the Degrees-of-Freedom Afforded by Multi- 
path? The various degrees-of-freedom afforded by mul¬ 
tipath can be used differently by application designers. 
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For example, if we consider wireless communication at 
the physical layer, the availability of multiple paths can 
be used for either multiplexing or for reliability [ |244| . 
Similar options can also be used at the network layer, 
and the network designers should carefully evaluate how 
to use the available diversities. 

• Decorrelaied and Disjointed Links/ Paths. To truly reap 
the benefits of multipath, the pluralism and diversity 
available through multiple paths should be leveraged. 
Care should be adopted to ensure the independence of 
the selected links/ paths. 

• Avoiding too much diversity. The idea that diversity works 
well in moderation but can backfire when overdone is 
well known in common folklore. We know that “two 
heads are better than one” and also that ‘*too many 
cooks can spoil the broth”. While increased path diversity 
typically results in better performance and more robust¬ 
ness, there is generally a diminishing marginal utility 
with increasing diversity, and too much diversity can 
even potentially cause performance degradation ]241| . In 
particular, coordination and management overheads (such 
as the control overhead of managing a large number of 
routing paths) can become prohibitively large with greater 
diversity. The implications of this need to be studied in 
the context of network-layer multipath solutions. 

£. Multipathing and Green Networking 

The problem of energy efficient multipathing is an important 
concern for battery operated mobile devices. While mobile 
devices can increase their throughput by striping their con¬ 
nections over heterogeneous networks, this comes at a cost of 
higher energy consumption. With the proliferation of mobile 
devices, the incorporation of energy efficiency mechanisms 
into the design of multipath solutions has become extremely 
important. Early work on energy-efficient multipathing has al¬ 
ready shown the potential for improving smartphone's energy 
consumption |245| -[ P^ . 

Apart from its importance for wireless networks, energy 
efficiency is also critically important for datacenter networking 
(which operates at the scale of huge warehouses, where 
even minor savings can add up significantly). More research 
needs to be conducted in the area of energy efficient “green 
multipathing” to use the power of multipathing while reducing 
the energy consumption on the Internet. 

V. Conclusions 

In this paper, we have surveyed the existing literature on 
network-layer multipathing. After describing the motivation 
and benefits of multipath in networks, and establishing that 
the future of the Internet is multipath based, we discussed two 
important problems that are related to multipathing, namely 
the issue of how to compute the multiple routes to be used, 
and how should these multiple routes be used by traffic flows. 
While our focus is on network-layer multipathing, our cover¬ 
age is holistic as we also discuss issues such as congestion 
control, flow splitting, and resource pooling (which do not 


fall neatly into a single layer). In addition to highlighting the 
main problems in network-layer multipathing, and describing 
the various approaches proposed, we have provided a broad- 
ranging survey of multipath protocols in different kinds of 
networks. Despite a vast amount of work on network-layer 
multipathing. many open questions remain; we conclude this 
paper by highlighting some important issues that require 
further investigation. 
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